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Introduction 

Th.e purpose o£ this report is to summarize the 
procedures, guidelines, and decision-maklng processes 
for a multi--media course development project in Leader- 
ship, Psychology, and Management. The project was 
funded by the Office of Education through the Navy 
Purchasing Office for work performed at U.S. Naval 
Academy, Annapolis, Maryland. 

This course development effort was one of three 
contracts granted with the intention of exploring various 
methodologies for developing a multi-media , individually 
paced course at a university level. It was intended 
that the three projects yield both innovative and effective 
courses and such appropriate documentation for each 
effort that the educational community would benefit 
from the labors and the monies that funded these proto- 
type course development projects. 

Each project centered on a different type of course 
commonly found in a university program of studies. The 
hard sciences were represented by Physics > A project 
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in Economics represented tLos^e university courses whick 
are less quantitative and rigorous tlian the physical 
or mathematical sciences. TLe soft sciences, which 
relied most heavily on sub/ective analysis, was repre- 
sented by an Introduction/to Leadership. Psychology ^ 
anj Management. The intention, therefore, of the overall 
effort, was to yield appropriate documentation and 
procedural guidelines ^or course development efforts 
across the spectrum of subject disciplines. 

This final report, as a summary of the effort to 
develop an inJiv idual Lzcd, multi-jncdia course in 
Le;ulor.ship, Psychology, and Management, represents a 
portion of the throe year documentation for the develop- 
ment effort by Westinghousc Learning Corporation. It 
is hoped that the enormous efforts of the project personnel 
to develop effective procedures for a course development 
model in a soft science course arc adequately reflected 
in the following pages. 
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PREFACE ' MODELS 



When one talks about a model, there is some danger 
that the lack of a clear definition of th^t term car 
cause misunderstanding. For purposes of this report, 
a brief discussion of models is introduce! here and 
some definitions of the models to be incorporated in 
this report, ■ In general, modeling can be thought of 
as being a representation of a system or a part of a 
system in some form suitable for demonstrating the 
way the operation or system behaves or may be considered 
to behave. Additionally, the model should represent 
a process or endeavor that shows the effects of those 
factors which are significant. Thus, while the model 
may not be a mirror image of the actual phenomenon 
in all respects, it does describe basic inputs, outputs, 
and operational characteristics. Where appropriate, 
it provides insight into critical environmental 
conditions. The representation that a model is cast 
in may be pictorial, descriptive narrative, qualitative, 
or purposefully approximate to reality. If possible. 
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it ma/ be mathematical and quantitative. Generally, 
many have a,, .^ t : think o£ a model in terms of 
some schematic drawing or flow chart representing 
discrete sequencing. of processes with clearly defin-- 
able and replicable input and output items. Still 
others view a model as a formula for a given opera- 
tions performance in such detail that the operation 
can be replicated with no difficulty. 

In making a decision to construct a model, 
one has a number of options to follow. One type of 
model is the overall process model . It tends to 
be qualitative and often provides a generalized, 
pictorial representation, a graphic overview of a 
o/stem. Often it serves as a convenient starting 
point for more quantitative details. A functional 
model may be a narrative description or outline of 
functions without indicating the methodology for ' 
accomplishing the results. A procedural model 
would describe in sequential fashion a program for 
all the steps that are involved in the operation of 
the system. A performance model tends to be quanti- 
tative in nature, allowing the designer to establish 
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performance criterion necessary to achieve the 
systems' goals. This type is quite common in engineering 
where voltage, strain, and stress are precisely 
detailed for all system components. Time models , such 
as PERT (Program Evaluation Review Technique) , sub- 
divides the total job into clearly specified sub-tasks 
into related time blocks. Finally, there is a cost 
model which often graphically projects development 
or operational costs for a given system. 

There are other variations to model building 
that are determined by their structure. They can be 
iconic or look alike, analog , or symbolic . Finally, 
models may be defined by the method employed to 
achieve a solution. In this classification is the 
analytic method ^ numerical method , or a Monte Carlo 
method . 

The reason for this brief excursion into a 
classification of models is to suggest the wide 
range of approaches to models, and to offer a caveat 
at this point that the various attempts to define 
the processes developed for this project are not 
always in concert with everyone's conception of a 
model. 
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With this reservation aside, one can proceed with 
the justifiable optimism that the details and discus- 
sion to follow will represent, with the greatest 
accuracy possible, the actual processes that comprised 
the developmental effort for this project* While 
much of it will be narrative and procedural in nature, 
it will offer those supportive charts and graphs 
which would facilitate a comprehension of the 
process. There are no quantitative inputs and 
the quantitative outputs are more appropriately 
discussed at some level of detail in reports listed in 
Appendix A. Our general aim for this model is 
that it be both functional and procedural. Where 
it omits details on exact methodology for accomplishing 
certain results, it is assumed that professional 
expertise of the reader can fill in the necessary 
gaps. However, where possible, the sequential steps 
involved in bringing this project to its very 
successful conclusion with all its products, will be 
detailed. Overall, this report will attempt to 
define, by any technique available, the significant 
processes and endeavors which yielded the final 
course system. 
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r PROJECT MANAGEMENT DESIGN 



Abstract 



This section introduces the project develop' 
ment model by concentrating on the 
organizational characteristics and 
general management procedures employed 
in the project. In doing so, it offers an 
overview of products for the entire 
project. Additionally, the efforts to 
implement a PERT system as an ancillary 
aid to management control, are discussed. 
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Introduction 

Current professional literature is profuse with 
extensive treatises on design studies as they relate 
to systems projects or computer operations. The design 
of any system demands a logical completeness, a consid- 
eration and integration of environmental constraints, 
and a keen eye for auxiliary components and subsystems. 
Much of the literature has technical areas as subject 
matter, thus fairly discrete or concrete operations. 
But there is little on social systems operations. While 
there is a swelling literature on social systems thinking, 
theorization, or conceptualization, little is seen on 
developing and implementing operating systems for social 
utility. Our purpose here is to outline the operation of a 
management system for a multimedia development project 
for a college level course in Leadership, Psychology and 
Management with the intention of illustrating the dynamics 
of product development effort. 

Initial Tasks 

The WLC proposal for the Management Psychology 
Course was awarded on 26 May 1968, and it can be assumed 
that one of the contributing factors in this decision 
was the sophistication of the design elements already 
incorporated and articulated In the proposal. 
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In addition to the precise integration o£ all subsysten^s 
for complete contract performance, there were innovative 
models that promised a major breakthrough for curriculum 
development - a media presentation and selection model 
that went beyond a simple or gross comparison of media 
to an examination and evaluation of those variables that 
are inherent to media's ability to interact with the 
learner. The model's premise was that media selection 
should be based on the appropriateness and educational 
worth of variables imbedded in each medium, not the 
medium itself. 

Even before announcement of the contract award, 
representatives from WLC began an extensive orientation 
to the subject area of naval leadership. Later, they 
would be aided by the guidance of the USNA project 
manager, a faculty member of the Leadership committee. 
Detailed memos were then distributed to the writers and 
analysts responsible for actual materials development 
at the Behavioral Systems Group in Albuquerque, iNew 
Mexico. Several visits for on-site inspection and 
orientation were arranged to give first hand awareness 
of the Academy and the environment of the students 
using the course materials. Films, books and 
appropriate naval service documents on regulations and 
procedures flowed between Annapolis and Albuquerque. 
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Concurrently, an office icas established in the 
city of Annapolis with the primary function of acting 
as liaison between the customer and the various agencies 
of the corporation that would contribute to the project 
effort. Secondly, the Annapolis office would serve as 
the systems center for data collecting and processing 
unit for the project. The initial task for this group 
was the definition of the data system that would operate 
for project evaluation and student data analysis. 
Management Model 

The first model or conceptualization of process 
to be discusse^ is project- management. This will 
deal with the overall mechanism which directed 
and controlled the vatious operations which yielded 
all the products of the project and the final course 
system. A suitable starting point is a brief 
detailing of the project output, which might give 
a feel for the magnitude and complexity of the 
project. Figure 1 represents one schematic of 
the tangible products which emerged from the project. 
It is labeled a product tree, since it attempts to 
give a structure to the various products as they 
emerged in time. However, like any analog, it is 
not a perfect match. It does, however, represent, 
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from a management point of view, the specific 
products wtici would Become the visible and demonstrabl, 
achievement of management's efforts. Appendix B 
provides a code and Brief details on each of the 
product numBers. although they are discussed in detail 
in other sections of this report. 

In realizing the generation of these products, 
the project staff „as organized along two lines. 
Figures 2 represents the project organization as 
It was generally maintained throughout the course 
Of the project. Like all organization charts, it 
is illustrative of the various position assignments 
and hierarchical structure of staff personnel only. 
However, it does indicate the general makeup of the 
staff by position. 

Table I follows immediately after, with brief 
summaries of the assigned tasks and function of 
those positions specified in Figure 2. 
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TABLE 1 

Project Personnel - Assigned Tasks S Function 



Project Manager 



Responsible for the over- 
all work o£ the project. 
Specifically functioned as 
policy maker, planner, 
director and quality con- 
trol for all phases of the 
contract. 



Project 

Coordinator 



Although in a staff position, 
the Project Ccc/dinator was 
primarily responsible for 
xhe day to day articulation 
between all phases o£ the 
project. 



Contrs.ct 

Administrator 



There was a double role o£ 
insuring contract account- 
ability as well as fiscal 
res];^onsibility as business 
manager. 



Materials 

Development 

Manager 



The prime responsibility 
was the development of 
effective learning materials 
from initial planning and 
development through the 
many revision cycles. 



Editorial 
Director 



Functioned in direct control 
for the development and 
revisi.:>n of manuscripts. He 
specified in detail the 
standards and specifications 
for style and editorial 
procedures. 
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TABLE 1 



riter/ 
Analyst 



In addition to the actual 
v:riting and revision o£ 
instructional materials, 
their repertoire included 
sufficient background in 
behavioral sciences to 
apply continual analysis 
o£ behavioral implication 
to the learning programs. 



Subject 
Matter 
Experts 



They provided not only 
military expertise for 
appropriate examples, but 
collectively represented a 
broad range o£ subject 
discipline knowledge in 
management, psychology, and 
leadership theory and practice, 



Audio-Visual 
Specialists 



The primary task of the 
Audio-Visual Specialist 
was to act as interface 
between the project personnel 
and the various outside 
consultants consisting of 
artists, TV production per- 
sonnel, and other audio- 
visual resource agencies. 



Production § 
Control 
Manager 



Responsible for the physical 
production of all materials 
and their conformity to 
acceptable editorial and 
educational standards. 
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TABLE 1 



Production 

Supervisor - Usually a chief editor, 

his job was to insure 
editorial standards and 
performance • 



Document 
Control 

Chief - This position grew out of 

a pressing need f^rr centra- 
lized control o ^ue many 
source dccumeiits -^at com- 
prise an ever-exr ::.ading 
editorial file consistently 
being accessed by all staff 
members. 



Data 

Processing 

Manager - The chief function was the 

des ign , development , and 
maintenance of the data 
processing system for 
developmental evaluation, 
research analysis^ and 
cost accounting. 



Analyst/ 

Programmer - Detailed and coded the project 

EDP system and maintained 
the data files as appro- 
priate. 



Research 

j Director - The prime role was the design, 

; analysis and on-going revision 

; of all research and evaluation 

; measures. 
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TABLE 1 



Customer 
Liaison § 
On Site 

Instructor - In addition to maintaining 

continual interface with 
the customer J he was charged 
with instructional res- 
ponsibility for the course 
under development to insure 
appropriate feedback on 
effectiveness at that level* 



Course 

Administrator - In conjunction with the On 

Site Instructor and the 
Research Director, the 
Course Administrator was 
responsible for the day to 
day operatj-i^n of the course. 



Project Development Functions 



Th-us, Table 1 provides a link between tke formal 
organization o£ the project, as^ sliown in Figure 2, and 
the eleven functional categories that evolved during the 
project. These functional categories and their rela- 
tionship to management is suggested by Figure 3. 

This representation together with the previous 
organization charts and tables represents the management 
model £s it evolved during Phase I of the project, and 
which has essentially remained the same throughout. 
The following rages are devoted to three discussions of 
the eleven functions which comprise the model. 

1. Project Management ; Functions which fall into 
this category cover all aspects of internal 
management and customer interaction. 

2. Course Design and Analysis ; This category 
contains functions relating to the systematic 
analysis of materials and the design of the 
course and objectives. More specifically, tasks 
included such activities as acquisition, study 
and classification of course materials, 
specification of course content, construction 
and sequencing of all objectives, interaction 
with subject matter experts, and coordination 
with the materials preparation function to 
insure student attainment of all objectives. 
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Presentation Design and Media Selection ; This 
category was partially unique to this contract 
in the sense that it is usually considered a 
sub-function of course design rather than an 
independent function. The development of a 
separate category was due to the research 
emphasis in this project. The function of 
presentation design was not only to specify 
the optimal presentation design and media for 
any given objective, but to do so within the 
constraints of the research design. Tasks in 
this category included classification of 
objectives > development of a theoretical 
rationale Tor specifying presentation design, 
developing an empirical appl i cat ion of this 
theoretical model, specifying the dimensions of 
presentation design and media selection to be 
investigated, and careful monitoring of all 
final course material modules to insure their 
compliance with original specifications. 
Materials Preparation and Behavioral Editing : 
I'liilOiiiiMM in I hi i'/M<'p,niy covpicmI nil /M'livitior* 
rt^lnlhiH I Im« ni'lnnl |M t^pnrn I Hmi nl' i iHii'nn 
materials* Specific tasks included construction 
oT all teflts, propnration of nil film scripts. 

Mi 
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slide sequences, lecture scripts, workbooks, 
texts, and support materials. The behavioral 
editing function was the most important aspect 
of materials preparation in that it insured the 
instructional effectiveness of the material. 
Behavioral editing was concerned with the 
continual monitoring of the stimulus response 

progression. Hach discrimination must occur in 
the appropriate sequence to insure that effective 
learning will take place. 

Research and Evaluation ; This general function 
covered the specification and design of the total 
research effort. Tasks included the specification 
of student data base elements, specification of the 
experimental design and statistical analysis, 
specification of data collection procedures, 
development of assessment instruments, and 
interpretation of experimental results in the light 
of existing research in learning theory and 
educational psychology. This function was unique 
to this contract and would not be included in a 
general model for materials preparation. 
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Materials Validation : One of the n.'cessary steps 
in the construction of all good instructional 
materials was the validation or empirical tryout 
and revision of materials. This functional 
category covers all aspects of the validation 
procedure. More specifically, tasks included 
implementation design, data collection, item 
analysis of progress checks, error diagnosis of 
individual frames, and collection and analysis 
of affective student and instructor responses. 
Production and Control : This function covered 
all aspects of materials production and^^qiiaj^ty 
control. Tasks include typing and copy edrtin 
of all reports and materials, insuring compliance 
with submittal formats, graphic and packaging 
design, printing and filming specifications, and 
all illustrations and art work. 

Project Coordination : This function would generally 
be included in the same category as project 
management. Because of the diversity of the 

wiiH. liol ioviul iluii II ;io|WH*<Ho ccHiril ttuil ton I tinci ion 
would be necessary. The functions cover general 
implementation and maintenance of management 
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decisions. Specific tasks included such activities 
as the establishment o£ project schedules and 
insuring their compliance, monitoring of production 
and submission of all reports, establishment and 
maintenance of complete filing and document 
retrieval systems, maintenance of the cost 
accounting system, and general interdivisional 
coordination activities. 

Cost Accounting ; Cost accounting was necessary 
for maximum efficiency in any project. In addition, 
the multimedia course development project requires 
the accumulation of costs by function and task. 
This is necessary to determine cost effectiveness 
of the various materials and of the model itself. 
Specific tasks include the design and implementation 
of the cost accounting system, development of 
computer programs to accurately isolate all direct 
and non-direct labor costs, and the development' 
of n floxiMo roo*ll)iu'k syj-.tom for nil WliC 
tlop'>«*t»KMilH iiivolvtui in lliu |MM)Joct orforc. Thin 
category may eventually be assumed under either 
project coordinnt iofi or general ndminlstrn ti on . 
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10. Ccn tract Administration ; Thds function was 
similar to tLat of any long-range contract. 
Included in this category vrere all tasks involved 
in insuring compliance with, the contractual 
agreement that has Been established with the 
customer and direct customer interaction on all 
aspects of the project effort which affect the 
cost, scheduling, or other contractual committments. 
Data Processing; This category actually functioned 
as support to many of the previously mentioned 
categories. However, because of the personnel 
and physical facilities required, this function 
was generally considered to be procedurally 
independent. Specific tasks include such activities 
as establishing a series of data bases and 
information retrieval systems for the research 
effort, for the validation of materials, and. for 
systems management. It also included implementation 
of materials utilizing CAI components. 

It should be emphasized that the above were 
functional categories. They do not necessarily represent 
responsibility assignments. Often one individual was 
assigned to two or more categories or vice versa. This 
distinction is important because it is the functions and 
procedures which can be generalized to similar projects. 
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This model was continually revised with respect 
to four criteria: 

1. actual functions performed in each, category 
compared to theoretical functions 

2. interactions of functions in each category 

3. interdependencies of functions in each 
category 

4. cost-effectiveness of the categories 
Manpower Requirements 

With respect to manpower types and loadings, Table 2 
indicates the various special manpower requirements that 
existed throughout the course of the contract. Its 
function is that it be illustrative of the complex man- 
power requirements given a development effort of this 
magnitude. For the management model, it represents a 
personnel resource table. 

In reviewing this table, one will note the use 
of Min/Max headings. This represents the minimum number 
of personnel employed at any given time in the course of 
the yearly phase, and the maximum number required at any 
given point to complete the various developmental tasks. 
The purpose in using the Min/Max notation is to illustrate 
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MANPOWER TYPES AND LOADING 
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th.e broad swing o£ personnel req^utrements in a given 
year for some types of expertise. On t?ie other hand, 
it illustrates the continuing requirement of other 
personnel types at a fairly constant level. 

In general, the model was quite stable over most 
of the project, and although, there were a number of 
personnel changes throughout the course of the three 
year project, the organization, as specified in Figure 2, 
and the functional activities, as specified in Table 1, 
remain constant. 

Other Management Responsibilities 

Another aspect of the management model centers 
on some specific responsibilities necessary for 
continuous project performance. They are generally 
divided into two areas: production responsibilities 
and environmental responsibilities. 
Production Responsibilities 

Instructional Materials . This involves the manage- 
ment of production and delivery of the materials 
generated by the project. 

Proposal Contract . This involves monitoring the 
product development against the proposal specifi- 
cations.' 

Product Quality . This involves responsibility for 
all aspects. £o the product quality generated. 
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Environmental Responsibilities 

Budget. This involves reviewing performance against 
projected costs, in form and figures. 
Staff Morale . This involves keeping everybody 
happy Cif that is their Best state) and willing 
to work. 

Interagen cy Cooperation . This involves seeing that 
the atmosphere between agencies involved is as 
good as possible for productivity. 
Staff Hiring. Staffing the project with people who 
will work efficiently. 

Intraagen cy Cooperation . This involves keeping 
those within the project in the same state aimed 
at in Staff Morale. 

Project A ccountability . This involves seeing that 
the project meets its commitments. 

These responsibilities are ordered in a network 
shown in Figure 4, which attempts to link those responsi- 
bilities just cited in graphic organization. Generally, 
these items are an attempt to delineate some of the 
more intangible aspects of the management effort. There 
is no attempt here to discuss those procedures which 
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resulzeC in zKo naintenarice of norale, cooperation, ore. 
Their listing, however, suggests tliat tiie manag eirient 
nodel must be concerned v/ith. those intangibles which 
grov; out o£ people \\^orlcing together. It should be 
e":*phasized, as one exarapie, that product quality is 
quire different from products per se, and that one of 
the functions of management is to delineate the various 
tolerance levels for acceptable performance in addition 
to specifying the overall criterion for a worthy product 
Ci.e.^ each product should not only be acceptable to 
the customer, but should exriibit rhose characteristics 
which iUakes it acceptable enough to bear the company 
hallmark without reservation) * 
General Product Planning and Coordination 

The work flow chart > Figure 5, represents a critical 
path work flow which was the basic management tool for 
insuring total project coordination of all product tasks. 
This work flow chart also served as the input to the 
PERT analysis and control unit discussed later in this 
section. 
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Time Frames and Schedules 

Any discussion of tinies frames, sckedules, and 
slippages at a detailed level ^^''ould liave no purpose 
for this management model. It is possible, however, 
to demonstrate the relative schedules or the various 
activities over a time span of three years. The 
schedules shown in Figure 6-1, 6-2, and. 6-3 
indicate the schedule as originally devised at the 
beginning of the contract. IVhile 'chere were some 
alterations to this basic, total schedule, in retro-- 
spect it still serves as a clear model for allocation 
of tasks on a temporal basis. These figures, in con- 
junction with Figure 1, indicate the relative relation- 
ship and sequencing of all phases of product development. 



1907 1968 196A 

NOV. 3 0EC.1 DEC. 29 JAN. 26 FEB. 23 MAR. 22 Af R. 19 MAY 17 JUNEUJUtYU AUG. 9 SE^T.O 



SpMlftCAtlon #f 





U2 


U3 


U4 


"5 


U6 


U7 







U2 


"3 


U4 


Us 


U6 


U7 


U0 



fttidli Anriyiit 



• fll llf IMW fllr MNW 

•ndUH 

In ^fMWi EViHictiwi 
Rctfifnv of ExMtfif 



froductlon of Count 



Ovilfln of AttM 
■no Kwuvtnm 

todfvTMit 
Qflfimto ItMi) Pool 



WMu Ooto fItductlOA 



I I I I I I I 



Ddfift foe IdtptoftMOMvlOA 





U2 


U3 


U4 


Us 


U6 


U7 


Us 



Ul 



U2t uJ U4 



I uj uj u j Us 



Throufli Ui>H 4 



Throofh OfUt S 



[ OoH>1onrf2 Wiij g«ilti3tnd4 



UntttStndS 



UnHt3 



UfiHsS 
•ndS 



|lMto1>.id2 



UAitt3ond4 



UoHS 



l2 l3Md U 



Is tod Is 



[•71 







T3»ndT4 


TstndTs 



lt«m Difficulty and Ittm Validity 



Figure 6^1. Temporal Allocation of Tasks During Phase I 

33 



oeT.4 »ov.i «o..» i»c.i> jSTm «A«.»»i..i< »Av i« ««l««t»n «»■• «"•• 

I I I I I 11 I I I I I 



«M«»MM«rM lor 



WmtfupportMMrM 



WWt# TflMi 



OittOitMattrUltfli 



FroductlMof Ceufi* i ii.«tfy- 



I iT 



.1 «»i «el 



«»| «J «»i 



Ovrtlop Comp«I«f frogrun (w Dm An»ty«i« 



□ 



Figure ^-2. Temporal Allocation of Tasks During Phase n 

34 



I 



•5- * 

t ' 
1 A 



if 



HfT.t OCT.J OCT.Jt HtfV.n OCC.H MN.ll «•» M*»l.»AfP t» M*V H JOtii W JOLV 10 AU0.7 SffT.« 

1111 I I I i — I — I — \ — L_ 



fM*lCtl«ll«fCMfM 



0(1$ dl Milfrlfll tntf 



^H**l f fOifuclion of TfMl 
•fid CoufM MMtliftI 



rintl Vahdoltofl of T«it< 



ERIC 



Figure '6 <'3. Temporal AUocaUon of Tasks During Phase in 

35 

46 



PERT Analysis and Control 

There was a concerted effort to test the feasibility 
of modern management tools for determination of: 

• time estimates for completion of any and 
all parts of the project. 

« resource requirements (manpower and other 
resources) for any and all parts of the 
project. 

o the effect of schedule shifts or delays on 
resources, project costs, and delivery dates. 

• the effect of variation in sequence or 
inclusion of project tasks on resources, 
schedules, and project costs. 

o on-going performance evaluation which could 
be used to continually readjust resources, 
time, and cost estimates for the project. 

It was decided that the PERT time system could 
best be adapted to form the basis of the planning and 
control systems for the multimedia project. 

The PERT system, initially developed in 1958, 
has proved to be a useful tool in planning and evaluating 
projects in development and construction. Most of these 
projects have been concerned with the development and 
construction of weapons and military materials. 

While the computerized PERT system could not be 
maintained throughout the project, the effort to detail 
the inputs into, the system served as the basic strategy 
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for isolating tasks, establishing their interdependency, 
and maintaining control on the project activities. The 
recommendation, therefore, is that PERT analysis is a 
vital element for a course model development and is 
discussed here in some detail. 

The application of such a system to any research 
and development project is based on the presumption that 
sufficient experience with the activity exists to permit 
the activities involved to be identified. Since many 
similar systems have been used extensively for efforts 
in technical writing and materials preparation, considerable 
information was available for the establishment of a 
large portion of the procedural steps involved in the 
project. On the other hand, many of the research 
activities were relatively new and no adequate systematic 
outline o£ procedures had been developed to serve as a 
model for the project. In general, developmental projects 
in education and projects which are primarily concerned 
with research in any field have not been reported in 
sufficient quantity to establish a generalized 
procedural model. 

It was believed, however, that a careful analysis 
of the work necessary for the accomplishment of each 
milestone would lead to the development of a general 
list of procedural steps. It is important to note that 
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regardless of how well (or ill) defined the work tasl s 
are, deviations from the general model will occur. 

Developmental activities are relatively new to 
the field of education. The problem involved in the 
multimedia contract was the production of a particular 
kind or type of product, from the initial planning 
stages through the development of prototypes which 
can be used for future production models. 

The research aspect of the project can be divided 
into three major areas: (1) evaluation, (2) experimentation, 
and (3) validation. 

The general nature of the evaluation research 
involved the use of existing data and the collection of 
new data to answer specific questions and hypotheses. 
Attempts were made to secure valid and reliable information 
from the USNA files, by the careful examination of 
historical records and contemporary progress reports. 
Additional data was collected by an extensive testing 
program at the USNA. In developing the network of 
activities associated with the development and compilation 
of student data, the development of the original hypotheses 
was among the first steps taken. 

The unique aspect of this project was the 
interdependency between the experimental design and the 
development of course materials. In order to evaluate 
the effects of various media and presentation design 




elements on the student *s learning style, it was necessary 
for the research design to set the parameters for the 
development of the materials. For this reason, interaction 
between the two major paths of the network was required 
in all phases of the project. 

The accuracy of any ma.iagement system is a function 
of the precision and clarity of the original work 
specifications. The total project must first be defined 
and then subdivided into a series of major units of work. 
Each of these units represent milestones of accomplishment 
toward the completion of the project. Milestones 
represent key network events that are necessary to 
achieve the total project objectives. 

The PERT system included two major cycles: the 
planning cycle and the implementation and control cycle . 
The planning cycle included: 
o definition of project objectives, 
o specification of milestones for the project, 
o breakdown of tasks. 

o further breakdown within organizational departments. 

o definition of resources. 

o development of a cost-accounting system. 

o construction of the networks. 

« estimation o£ time and resources for each task. 

o development of delivery schedules for all milestones. 



1 

5tf 



! 



i 

$ 

3 

i 



j 

1 

I 
I 

1 

T 

f 



The implementation and control cycle includes; 

^ development o£ all data processing requirements, 

• implementation of cost -accounting system. 



resources to projected expenditures, 
o updating of PERT network. 
^ revision of schedules and budget. 

1. Defini£ionjo£ P^oj£Cit 2?^ j.^ctiyes__ 

The first step in developing the PERT management 
system was to define the objectives of the project. 
These objectives included -1^ statement of all 
deliverable documents for the contract. To accomplish 
this step, a careful review was made of the proposal, 
the "Multi-media Course Development Paper", the "Program 
Data Book", and all other written and verbal communications 
between WLC and the USNA. 

The multimedia course development project had 
three major objectives: 

a. The development of a multimedia course in leadership 
for the USNA. This course was to include the most 
recent advances in the field of educational 
technology and was to be fully validated in the 
classroom. 
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comparison of actual expenditures in time and 
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b. A comprehensive research program to evaluate the 
effectiveness of various media and presentational 
forms with respect to variables including cost, 
student achievement, and administrative difficulti 
Relationships between individual learning styles 
and subsequent performance in the instructional 
environment were to be carefully studied. 

c. The primary long-range objective was the design, 
development, and verification of a model which 
could be used for the development of a superior 
educational system. 

2. Specijf ication of_Milestones 

The major milestones were initially defined as: 

(1) the research and evaluation plan, 

(2) the content outline, 

(3) the terminal objectives, 

(4) the enabling objectives and test items, 

(5) the presentation design, 

(6) the segment documentation packages, and 

(7) the actual material. 

In addition, progress reports were to be submitted 
monthly and quarterly. Additionally milestones were 
believed to be important for the fulfillment of the 
contract objectives, but were not part of the critical 
path for the workflow of course construction and the 
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research effort. These miler tones include: 

(1) the development of a cost-accounting systeiji, 

(2) the development of a rationale for media selection, 

(3) the development of the rationale for sequencing, and 

(4) a series of technical papers which included topics 
such as course description, course strategy, course 
development model, and cost effectiveness. 

3. Brejakdown o^JTasks^ 

The specification of tasks served as the basis for 
construction of the PERT network* The actual task 
breakdown began with the construction of the course 
development model . The development of this model and 
the task definition began at the most general functional 
level. The subdivision of the functions continued until 
a level was reached at which the complete resource 

i 

requirement could be completely specified. All tasks 
necessary for the accomplishment of each milestone were 
delineated. Care was taken to ensure that these tasks 
were consistent with the work statements presented in 
the original proposal. 

After the tasks were specified, their 
interrelationships were defined. 

Up to this point, the relationships between the 
course development model and the structure of the PERT 
network were quite similar. At this point, however, it 
was necessary to assign responsibility for the specific 
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tasks to departments or units of operation in the 
customers* organization. From this point on, the 
development o£ the PERT network began to reflect the 
organizational structure and specific resource 
availability of the customer and, therefore, deviated 
from the general model. 

4 . Ta£k_Breakdown_Wi^thin Or£ani£ational_Department^s_ 
Each department in the organization identified a 
number of steps within any task that was assigned to 

it. 

Another variable which influenced the extent to 
which the steps in any task were identified was che number 
of departments and different personnel in the contractor's 
organization which were responsible for the accomplishment 
of that task. For example, the drafting of a report was 
usually broken down into two smaller steps editing 
and typing for assignment of responsibility to 
different sections. On the other hand, the development 
of test norms, which was a much larger task, was not 
broken down since the responsibility for the various 
steps involved rested with a single section in the 
contractor's organization. 

As the structure of the step breakdown is defined, 
the interfaces between any of the various steps should be 
identified. Interfaces are defined as the transfer of 
information from one part of the network to another. 
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5. De£inition_o£ Resources 

Closely related to the breakdown, with respect to 
organizational structure, is the definition of resources. 

These categories included: editor, secretary, 
project coordinator, writer, analyst, illustrator, 
research assistant, research associate, production 
and control manager, and subject matter expert. For 
each step in the network, the number of man hours 
required for each resource was specified • 

After the specification of resources and man 
hours had been completed, manpower loading for any given 
project-day was determined. Charts showin^j the overall 
requirements for each of the professional resources 
were developed. 

These charts were analyzed by the management 
of each department to shift resources in order to 
alleviate the problems of excessive manloading or 
manpower. Highly irregular loading patterns would 
indicate a need for additional hiring, overtime, or 
possibly a rescheduling of the activities listed in 
the PERT network. 

6. Dev^elo£ment of_ a Cps^t^Accounting^ System^ 

One aspect of this system which was alluded to, 
but not presented in its entirety, was the relationship 
of the cost-accounting system to the PERT network. 
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The code structure for the accounting system was 
devised in such a way that costs could be isolated 
for any professional resource (limited to the ten 
previously mentioned) working on any step in the 
network. This aspect of the project was necessary 
for the feedback, updating, and control of the system. 

All costs could be summarized with respect to any 
given milestone, resource, or organizational department. 
In this way, the actual man hours spent for any activity 
could be directly and immediately compared to the 
projected hours for that activity. Variations in the 
actual and the projected man hours served as a basis 
for revisions of projected costs and milestone schedules. 
7. Construct ion_of the Network 

The network was the graphic representation of the 
project plant It showed the plan, which had' been 
established to reach each milestone, along with the 
interrelationship and interdependence of the project 
steps. 

A network was composed of events and activities. 
Events is the term used when referring to the start 
or completion of an activity. These do not consume time 
or resources. Events are points in time which signal 
the initiation or completion of an action. 
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8. Estimation j)£ Timejandjlesource Requirements 

As previously mentioned, the resources or man 
hours were specified for each step or activity in the 
project effort. This, of course, directly effected 
the times which were assigned, but did not completely 
determine them. 

Once the time estimates for each step were 
determined, the longest time path through the network 
was calculated. This path is called the critical path, 
since the total project time is dependent on this path. 

Although only one estimate of time was used, the 
fixed beginning and termination of the contract allowed 
for the computation of slippage time. The time required 
for the completion of the critical path was calculated 
forward from the initial date and backwards from the 
termination date of the contract. This calculation 
yielded the maximum amount of "slippage" time that 
could occur without affecting the final delivery date, 
as specified by the contract. 

9 . Development of J)el iyer^ Schedules^ 

By analyzing the manpower loadings and the total 
worktime required for each activity, the delivery 
schedule from each milestone was developed. Initially, 
three times were indicated for delivery of each milestone. 
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They were: 

• delivery to USNA. 

• return to WLC with recommended revisions. 
« final delivery to USNA. 

All three times were indicated on the schedule 
because the amount o£ turn-around time allotted for the 
USNA to review materials was critical to the completion 
o£ the contract. 

The PERT management network represented an ideal 
system for the planning and control of the project. 
During Phase I, it became increasingly evident to WLC 
that the use of the PERT system for allocation of 
resources and estimation of deadlines was decreasing in 
effectiveness. After a careful examination of the 
project, there appeared to be three factors which 
continually caused problems in the management of the 
effort and eventually led to the formal abandonment of 
the PERT/cost system, although the analysis had yielded 
the basic methodology for management that would remain 
throughout the project. 

1. A management system such as PERT assumes that all 
activities can be assigned reasonable estimates of 
time and resources. It also assumes that all 
activities can be brought under the control of the 
contractor by manipulating the resources or time 
involvod, in most dovo] opmontnl projorin where 
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PERT has been used successfully, the interaction 
with the contractor, i.e., review points, occurs 
at the termination o£ a long chain o£ activities* 
In a materials development project such as the 
multimedia project^ continued interaction with 
the contractor and "small step" approval was required. 
One way to look at this problem is to consider the 
difficulty that would occur if one department in an 
organization was not scheduled by the PERT system but 
had a vital part in the development of the materials. 
The requirement for continual interaction with the 
customer and multiple sign-off points made the accurate 
specifications of the customer's time on the project 
essential. Without this control, the careful scheduling 
of resources and time around undefinable blocks of time 
was not feasible. 

In order to efficiently schedule a project of this 
type, a contractual agreement would need to be established 
which would specify the amount of total turn-around time, 
including specifications, necessary for each submittal. 
The establishment of such events, prior to the project 
start, would be advantageous to both the customer and 
the contractor. 




A second problem which served as both the major 
reason for the development of the PERT system 
and the major reason for its formal abandonment was 
the milestones. In almost every case, the approval 
of one milestone was necessary before the work on 
the following milestone was begun. This requirement, 
in combination with a limited time deadline, posed 
a major problem for both the contractor and the 
customer. For example, the course outline served as 
a basis for the terminal objectives, which in turn 
served as a basis for the evaluating objectives, etc. 
on through the presentation design, segment 
documentation, and materials preparation. Therefore, 
the approval of the course outline was essential 
for work to progress from the contractor's point 
of view. The customer, however, was hesitant 
in making a hasty decision on a document which 
would serve as a basis for the entire project. 
This problem was never resolved. Consequently, 
milestones were not approved and dependent 
milestones were delayed. The first milestone 
of the project, the content outline, was 
InfornuiUy siibm i i r.tnl in ^^^!,^•^v., 1968, and aftor 
multiplo revisions, it has not yot boon coniplotoly 
opprovod, tts of tho writing of this report, 
AprH, 1971. 
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When this type o£ dependency relationship is inherent 
in a project, an agreement should be reached between the 
customer and the contractor as to the maximum time 
allotted for approval and revision, of critical milestone 
delivery dates. 

-3. - The management system developed for a specific 

project is a function of the precision and clarity 
of the original specification of the work to be 
completed. Because of the lack of clarity of 
the initial contract, the PERT network was based 
on milestones which the contractor believed to be 
the best representation of the customer's requirements; 
however, agreement was never officially reached as 
to the items which would constitute total delivery 
for the project. This lack of detail on project 
products, which formed the basis for the PERT 
network, caused major changes in the actual path 
of the system. Since this type of change could 
not be handled by the data processing program, the 
man hours required for rescheduling all activities 
became prohibitive. The major value of the 
computerized system was lost when deviations from 
the initial path were so extensive that they could 
not be automatically updated. 

The solution to this problem is simple in theory, 
but difficult in practice. A clear statement of the work 
to be accomplished and the milestones to be reviewed by 
tho customer should bo included in the contractual agreement. 
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This brief summary o£ management design consider- 
ations has attempted to introduce the development model 
by indicating the basic framework of operations. Since 
the management aspects of the project are difficult to 
discuss separately from the substantive operations, it 
is hoped that the management requirements for each step 
of the developmental effort vrill become more increasingly 
clear in successive chapters. Nonetheless, this chapter 
on management aspects of the project suggests that de- 
velopmental course systems are costly in manpower, which 
always absorbs the greatest portion of any budget. 
Secondly, that the complexity of the operations requires 
managerial skills that will effectively channel all 
the diverse technical competencies. 

Lest these considerations seem too formidable 
for those who would wish to replicate the process for 
their own development efforts, it should be noted that 
this project functioned as a prototype to explore and 
analyze developmental procedures in detail. With the 
lessons learned, or more precisely, with the map 
charted, replication of the effort for some other course 
is possible within a smaller time frame, and with reduced 



62 

51 



costs. 

Tlie competencies, ftovrever, can never be reduced 
or taken for granted. And management competencies are 
absolutely vital to success of a course development 
project. 
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II COURSE CONTENT BEFINITrON 



Abstract 



This section of the report discusses 
the procedures and problems in defin- 
ing content for the course in an Intro- 
duction to Leadership^ Psychology^ and 
Management , It concentrates both on 
the evolution of the content definition 
process and the contending points of 
view which made this portion of the 
course development activity the most 
troublesome. 
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II. COURSE CONTENT DEFINITION 

The course, An Introduction to Psychology and 
Leadership , is an introduction to the t^eory and 
techniques of Naval leadership developed through the 
principles of psychology and management. It presents 
the principles of the behavioral sciences which are 
pertinent to understanding individual ana group behavior 
of adults. It introduces midshipmen to the management 
process and the relationship of management functions 
to leadership. The course presents the generally 
recognized techniques of effective leadership which 
tan assist the Naval officer in analyzing, predicting, 
and influencing the behavior of his subordinates. 

The overall objectives established for the course were: 

1. to develop in midshipmen a knowledge of those 

principles of the behavioral sciences which are 

f 

pertinent to an understanding of the individual 
and group behavior of adults, and to develop an 
awareness of the applicability of these principles 
to Naval leadership. 

2. to teach the generally recognized techniques of 
effective leadership which can assist the Naval 
officer in analyzing, predicting, and influencing 
the behavior of his subordinates. 
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3. 


to introduce midshipmen to the management process. 




and to inculcate in them an understanding of, and 




an ability to use, the human relations aspects o£ 




that field as it relates to Naval leadership and 




to the duties of a Naval officer. 


4. 


to develop in midshipmen the ability to use the 




fundamentals of psychology, management, and 




leadership in the analysis and solution of leadership 




problems . 


Topics jCovered_in the Previous_Course 




The following is a list of general topics covered 


in 


the previous Introduction to Psycholoev and Leadershiu 


course. 


1. 


What Psychology Is and Does - Behavior as Adjustment 


2. 


Development of Behavior Leadership Growth 


3. 


Clinical Approaches to Personality 


4. 


Psychometric Approaches to Personality - Basic Statistics 


5. 


Psychometric Approaches to Personality 


6. 


Defining and Measuring Learning and Retention 


7. 


Management of Learning and Retention 


8. 


Observation and Action (Perception) 


9. 


Thinking and Deciding 


10. 


Motivation and Drive 


11. 


Emotion 


12. 


Reactions to Frustration 


13. 


Mental Health and Therapy 


14. 


Leadership and General Order 21 
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15. The Individual and the Group 

16. Interpersonal Communications 

17. Effective Leadership 

18. Leadership Basis 

19. Leadership Behavior and Styles 

20. Authority and Responsibility 

21. Cohesion (Morale, Esprit de Corps, Discipline) 

22. Introduction to Management and the Management Process 

23. Planning and Objectives 

24. Organizing and Coordinating 

25. Directing and Controlling 

26. Counseling - Interviewing 

27. Personnel Evaluation 
Goals of the Course 

Thus, the goal of the course was to teach 
midshipmen the skills and attitudes necessary to lead 
men effectively. These skills were: 

1. to plan the mission, assignment, or operation 
effectively, using good management practices. 

2. to realize that effective leadership depends 
on the credible image of the leader. 

3. to accept moral responsibility for the welfare 
of the Navy and the United States (special 
trust and confidence). 

These broadly stated objectives are quite similar 
in tone to the mission of the Naval Academy and to the 
goals stated in General Order 21. 
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By the nature of the contract, the experimental 
course would differ from the existing course in its 
presentational form. Aside from this, the emphasis 
of the new course would differ in several kjy ways 
from the present course. The differences were stated 
as follows: 

1. The experimental course would place a heavy 
emphasis on the 41 Critical Behavior 
Categories of NaVPERS 9224A and the film, 
Genercl Order 21. 

2. The constituent concepts of leadership would 
be derived from management and psychology 
principles, rather than merely related to 
management and psychology. 

3. The thrust of the new course woula be toward 
visible behavior applied, rather than toward 
simple cognition. 

4. The now course would require that midshipmen 
apply the rules and principles of leadership 
to solve problems. 
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5. The experimental course would use mixed-media 
to facilitate the simulation of real life 
situations . 

6. The course would allow the student to obtain 
the objectives on an individually paced 
schedule. 

In developing this course, the contractor took 
the following perspective: 

Effective Naval leadership is based on the 
successful application of psychological and 
management principles through appropriate 
behavior of the leader during times of 
emergency, during periods of training, and 
during everyday interaction with senior and 
subordinate personnel. Leadership is inferred 
from the visible, and often measurable behavior 
of those being led. When leadership is present, 
men will behave in ways quite different than 
when it is absent. Effective leadership allows 
the leader to predict and control the behavior 
of individuals within the group. An effective 
leader must be able to identify and diagnose 
instances where leadership has its own effective- 
ness in directing men toward the accomplishment 
of the Navy*s mission. 
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Initial Course Outline 

The content for the experimental course had been 
tentatively divided into eight major areas. These areas 
were outlined as follows: 



1.0 Characteristics of Individual and Group Behavior 

1.1 Science and technology 

1.2 Psychology as science 

1.3 Behavior 

1.4 Laws of behavior 

2.0 Self-Direction, Self -Motivation, Self -Control , 
Image 

2.1 Naval expectations 



2.3 Attaining goals 

2.4 Setting goals for yourself 

3.0 Supervision and Motivation of Subordinates 



3.2 Leadership psychology research 

3.3 Procedures for motivating subordinates 
4.0 Discipline and Morale 

4.1 The concept of discipline 

4.2 The concept of morale 

5.0 Relations with Superiors and Peers 

5.1 Formal 

5.2 Informal 

5.3 Rationale 



2.2 Self -expectation 



3.1 Leadership and supervision 



5.4 Techniques 



I 

1 




6.0 Organization, Selection and Administration 

6.1 Management - an overview 

6.2 Organization 

6.3 Selection 

6.4 Administration 

6.5 Decision making and problem solving 

7.0 Communications, Counseling and Personal Relations 

7.1 Models of communication 

7.2 Types of communication 

7.3 Personal relations 

7.4 Counseling 

8.0 Training and Guidance 

8.1 Learning 

8.2 Factors affecting learning 

8.3 Assessment of deficiencies 

8.4 Resolving deficiencies 

8.5 Presentation methods 

A more detailed breakdown of each of these content 
areas is in Course Outline I, Appendix C. 

For each of the eight major content areas, or 
chapters, general objectives had been developed as 
follows : 

Chapter 1. Midshipmen will understand that the knowledge 
and application of certain psychological principles 
are directly related to controlling the behavior 
of subordinates. 




Chapter 2. Midshipmen will understand that the basis 

for a Naval officer's behavior is naval tradition, 
naval custom, naval regulations, and naval 
standards of performance. 

Chapter 3. Midshipmen will be able to use the concepts 
of behavior mentioned in General Order 21 to 
accomplish the Navy's mission (including 
ramifications of good management, personal example 
and moral responsibility). 

Chapter 4. Midshipmen will be able to apply principles 
of behavior, management, and leadership to the 
establishment and maintenance of discipline, 
morale, and esprit de corps. 

Chapter 5, Midshipmen will be able to establish and 
maintain good personal relations with seriiors 
and peers on the basis of prescribed parameters 
of officer behavior. 

Chapter 6, Midshipmen will be able to employ men and 
materials efficiently using principles of 
organization, administration, and planning. 

Chapter 7, Midshipmen will be able to communicate 
effectively up and down the chain of command, 
in counseling with his subordinates, and in 
interactions with peers (laterally) , 

Chapter 8, Midshipmen will be able to use principles of 
training and guidance to erase performance and 
learning deficiencies among subordinate personnel • 
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Conflict of Views 

After official approval of the Content Outline I, 
with. tLe rider tILat certain attached comments be 
included, Terminal Objectives vere prepared and 
j subsequently submitted. A sample is furnished in 

Appendix D. 

1 With the submission of the Terminal Objectives 

was included a two-part supplement: one showing the 
relationship of the Terminal Objectives to the script 

; of General Order 21 and the second showing the 

relationship of the Terminal Objectives to the 41 

I Critical Behavior Categories of NAVPERS 9224A. 

The Terminal Objectives, which had been derived 

1. from the hierarchical structure of the Content Outline, 

j Csee Appendix E) , was rejected on the basis that most 

i 

of the Terminal Objectives did not adequately reflect 
j the ^'critical behaviors*' of NAVPERS 9224A and the 31 

critical factors of G021. 

I 

K The customer then proposed that the critical 

I behaviors from G021*and the 41 critical factors from 

NAVPERS 9224A be combined and that the 
I Terminal Objectives be deprived from this structure* 

This requirement brought on the first serious conflict 
1 of view between the agencies, 

I The development contractor stressed that it was 

in agreement with the customer in using, as the core 
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concent of the Leadership Course, material abstracted 
from the film script of General Order No. 21 and the 
41 Critical Behavior Categories contained in NAVPERS 
9224A. The difference of opinion was seen as not 
about the nature of the content; it was more specificallj 
concerned with the development of the content into a 
well-organized and properly sequenced course of 
instruction. 

Tho customer hud decided the Leadership Course 
should 1)0 constructed as a sequence of 81 "critical 
factors." liach factor was also categorized into one 
of oiKlH- chapters, which were themselves sequenced 
as the basic course outline. (See Course Outline II, 
Append i X C) 

Porhnps tho most serious difficulty envisioned 
wilh tl.iM proposod co.itont outline wns that it imposed 
» •Tlni" .miMiclnro on the course sequence. Every 
chapter and critical factor is independent of each 

^" tl.o scMw;o ,l,al tho loarnin,. of any particular 
segment is not based upon the learning of any previous 
segment. To say it another way, learning of one critical 
factor would not directly result in mastery of those 
subordinate competencies which are necessary to the 
acquisition of learning in subsequent segments. 
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The contractor contended that a course sequence 
relying on the 81 "critical factors" ran counter to 
Briggs (1968, Page 11): 
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"...This phrase (a flat structure) is 
employed to describe the organization or 
structure of the course when it does not 
appear to matter in what sequence the 
instruction for the various major 
objectives or sub-objectives is arranged. 
For a course with a flat structure one 
could either conduct the instruction in a 
random sequence or in any arbitrarily 
chosen order which one prefers. The 
significance of such a structure would 
be that the competencies gained in reaching 
each objective are independent from the 
compejtencies gained in reaching all other 
objectives. It is conjectured that if a 
course is well analyzed for the purpose of 
deciding upon the sequencing of the 
instruction, a truly flat structure will 
seldom be encountered. When one does en- 
counter an apparently flat structure of 
a course, one can question the value of 
the course, or perhaps the objectives 
should be restudied to make sure that 
they are in fact behavioral objectives 
rather than content objectives..." 

One basic problem in using a flat structure was 
that if the various parts of the content are related 
(in a "natural hierarchy") , then a forced flat struc- 
ture introduces a high degree of redundancy in the 
instructional sequence. The result was that associa- 
tions which exist between common elements are never 
made, and the logical connections which relate general 
principles to specific events are never formed. The 
task of learning is thus relegated to a kind of simple 
rote learning exercise, where only memorization of 
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discrete^ unrelated pieces o£ information is required. 

This, liLC «9ntended,, results in the absence o£ 
general mediae cs which would otherwise guide the 
learner through relatively long instructional se- 
quences. These mediators are the bases on which the 
student assimilates bits of information to form mean- 
ingful conceptr and principles, Ausubel, 1960 /as 
cited. 

The value of these mediators seemed well es- 
tablished in the experimental literature. Tosti (1967), 
for example, has shown that long-term retention is 
facilitated by the employment of mediators during ac- 
quisition. 

As further documentation, the contractor cited 
a consulting report submitted to the USNA, Briggs and 
Tallmadge (1^67) which described the necessity of 
studying and analyzing the content and objectives "for 
the purpose of determining the sequence of instruction. 
Included in this analysis was a detailed specification 
of the 'prerequisite competences', referring to the 
subordinate skills* and knowledges which must be ac* 
quired in particular sequences in order to attain 
the mastery of a particular behavioral objective." 

These and other references (e.g.. Bloom, 1965; 
Gagne, 1956; Mager, 1961; Gagne and Paradise, 1961) 
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seemed to agree that one stage in the organization o£ 
an effective Leadership Course would be to specify 
those behavioral objectives which were functionally 
related to each other. A number of references sug- 
gested that a hierarchical structure should be ex- 
pected after the content had been examined. Briggs 
(1968, p. 107) stated the matter: 

"..•It may be said that a number of studies 
have supported the view that many courses or 
portions of them, when analyzed carefully, 
display an inferred hierarchical structure 
whose validity can be supported by comparing 
sequences of instruction designed to follow 
the hierarchy with some other strategy of 
sequencing. Several experiments provided 
various kinds of evidence of the validity of 
the hierarchy, because when learning was se- 
quenced accordingly, learning progressed 
better than under other sequencing procedures. 

Hence, it was the contention that material to 
date was a well-organized and properly sequenced con- 
tent development, based on a number of contemporary 
theoretical and technological models. 

It was also emphasized that all 81 critical fac- 
tors were covered by the content outline; only the de- 
velopment, ordering, and structuring of the content 
was different. 
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The attempt to develop Critical Factor 17, 
designated "A good leader recognizes that his behavior 
affects those around him/' seemed to show the diffi- 
culty in the approach as once specified. It was 
criticized on the following grounds: 

1. The critical factor is not definitive enough 
to allow the writers to specify those 
events which are related to this critical 
factor. 

2. The critical factor is of such a general 
nature that several kinds of events could 
be related under it, as well as under other 
critical factors. 

3. The above condition, i.e., lack of speci- 
fication of a particular subject content 
or set of objectives which are appropriate 
only to this particular factor, makes it 
difficult to define the segment, leading 
to the inability to construct test item:> 
which in fact truly measure achievement 
within this segment, independent of all 
others. This difficulty prevents signifi- 
cant research to be conducted. 

4. Since the terminology used in the critical 
factor is not that used in any reference 
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source to identify content, it is virtually 
impossible to proceed from "terminal" ob- 
jectives to "enabling" objectives without 
complete dependence on the Subject Matter 
Expert (SME) as the main source of informa- 
tion. 

Examples of this latter point are as follows! 
In developing the construct, "A good leader recognizes 
that behavior affects others," the customer found it 
necessary to be more specific. He first designated 10 
conditions, which specified further factors and sub- 
factors involved in "recognition that his behavior af- 
fects those around him." Still, the subf actors, as 
indicated, are not sufficiently objective to be immedi- 
ately translatable into b-havioral objectives. For 
example: In attempting to interpret behaviors from 
the subcritical factor, "master your emotions," one 
would have to ask the follov;ing questions. Are we go- 
ing to give the subject techniques of mastering one's 
emotions, or are we going to say that mastering emotions 
is "good," or are we going to ask him why he should 
master his emotions? Should he be taught to discrimi- 
nate between the conditions where he can or cannot 
master his emotions? What types of emotions should he 
master? Enthusiasm, joy, pride? Does the term master 
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mean not display ? The statement of the subfactor leads 
us to many further considerations of what specifically 
is desired by this particular item. However, this is 
as expected. Normally, the analyst attempts to de- 
termine precisely what behay^ioral objective can be 
generated by taking such statements from the SME, and 
by such query, determines the specifics from the total 
range of those which might be generated. 

As a further consideration. Item 3 states, "main- 
tain an optimistic outlook." Obviously, if we asked a 
student before he even entered this course if a leader 
should maintain an optimistic outlook or not, he would 
quite probably predict that he should maintain an opti- 
mistic outlook. Therefore, the statement of condition 
of optimistic outlook is not sufficient. Should the 
course then teach him how to maintain an optimistic out 
look? Should the course in fact devote any time to 
teaching someone this "behavior", and if so, what? SME 
further specifies what is meant in "maintaining an op- 
timistic outlook" under a number of subcategories. 
These include "the will to win being infectuous," "cap 
italize on your unites capabilities and successes," 
maintain an aura of outwajrd calmness," etc. Here, per- 
haps is a level at which analysts can begin to develop 
some kind of terminal objectives and perhaps some kind 
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o£ content specification after much more interaction 
with the SMB. 

The second major difficulty, that of ''indepen- 
dence" of C.O. for. both research purposes and instruc- 
tional purposes, can be seen in the subf actor which 
states: "Be at all times physically fit, mentally 
alert, well groomed, and correctly dressed." Diffi- 
culty in "independence" is evident with this item. For 
example, this exact terminology was used in other Criti 
cil Factors. If, in fact, the course did a thorough 
job of teaching the value of being physically fit, men- 
tally alert, and well groomed, etc., in one segment, 
the course would be repeating in another unit and 
therefore expect relatively little change in behavior. 

The previously discussed subcritical factor, 
"master your emotions," also appears as another Criti- 
cal Factor. If the developers took this as a subf actor 
within the present Critical Factor, the course would 
again be teaching something which had already been es- 
tablished. 

The development of test items pointed out another 
problem, i.e., trying to devise content which will 
allow us to obtain this test performance without 
specifying some rule for mediation like "always be 
positive," or establish behavior regarding the "con- 




sequences o£ being negative/* etc. I£ the course just 
tried to teach this item in an inductive fashion, the 
student would have responded to anything which was es- 
sentially positive. This tendency to "pick the posi- 
tive" would have come under the specific stimulus con- 
trol of the problem types employed in instruction, and 
may reduce the tendency to transfer as much as just 
presenting the rule with no application. Generally, 
for good transfer, mediational rules should be used 
in solving problems. This being true, then stating 
the rule is as legitimate as a terminal objective as 
problem solving. Mediation is not only related to 
transfer, but to retention. Stating what the effect 
of being negative as opposed to being positive is, 
i.e., adding "meaningful" associations, wor.id more 
likely produce a better retentior situation. 

Re solution of Basic Framework 

Most of these points were made during a fruit- 
less four month treadmil) wherein test items and ob- 
jectives ^ere being generated ancj^ measured according 
to the j;iew content outline that relied exclusively 
on the critical factors. With less than four months 
remaining for Phase I, a new content outline structure 
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wa:* presented to the contractor by the customer agency. 
CSee Content Outline HI, Appendix G.) It basically 
returned to a reliance on content or subject ma. tar 
orientation (group dynamics, individual psychology, 
management principles, etc.) It proved to be eminently 
more workable since this outline, based on subject 
matter disciplines, had terminology and concepts whi h 
were much more concrete. Thus, it was able to avoid 
a dependence on vague and multi-meaning teriris, such as 
loyalty, courage, and the like, which was the basis for 
Content Outline II Cthe critical factors). 

Thus, Content Outline III CAppendix jS) became the 
basic framework for content definition for the remainder 
of the contract. The process whereby details within 
each part and segment were refined, was as follows. The 
content for individual segments of instruction were defined 
initially by the contractor and reviewed by the customer, 
who made further specifications. These specifications 
were then incorporaled into the course content outline 
for approval by the customer agency. Since this development 
effort for the course content outline was exhaustive. 
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finely detailed, and the product of much, reading, con- 
sultation, and discussion, it was assumed that upon 
final approval in Phase I, that content specifications 
for the course were essentially complete and not subject 
to revision. However, changes in specifications for the 
course outline were to be a constant problem over the 
next two years. Various agents for the customer required 
changes, modifications, or substitutions for what was 
considered appropriate and meaningful for the course. In 
almost every case of respecif ication of content, the 
customer basis for requiring the modification was a 
rethinking on appropriateness and emphasis not based on 
student performance data. 

The implications for the development models should 
be obvious. It had be^n the initial decision' in this 
project to start from a content base rather than a terminal 
objective base. Hence, from the content outline, terminal 
and enabling objectives were generated and the instructional 
materials and test item pool fabricated. <2ach change to 
the content outline, therefore, had a falling domino 
effect in that ahaiiges were required for every project 
item which grew out of the content outline. The unwill- 
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ingness o£ th.e various customer agents to give final 
approval to tdiat vas desirable and appropriate as course 
content outline, vas tHe major conflict of vievs area 
for the entire, project. Approvals transmitted in 
Phase I were denied any validity in later phases and 
were not honored by the customer, who claimed them to 
be merely provisional. 

Because of the experiences of this particular 
project in defining course content, it is very difficult 
to generalize on what the optimum procedure should be 
for that task in the model. In retrospect, it would 
seem that the procedures initially followed would be 
adequate, provided that at some given point in time, all 
parties to the development effort agree that content 
definition is fixed and subject to change only when 
student performance data indicates that some false 
assumptions were made about content and its relation 
to student capability. 

These operational procedures for content definition 
began with a general blocking out of major subject matter 
areas after a review of the appropriate literature and 
existing courses. Individual teams ^ composed usually of 
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a writer /behavioral analyst and a subject matter expert, 
drew up drafts which, were reviewed by an in-house staff. 
Tae product of their discussions were then forwarded to 
the customer where further discussion ensued after a 
review. The major discussion area and problem area was 
generally the requirement to mesh subject matter material 
with the requirements of a course for the naval service. 
On this point, discussions were lengthy and generally 
the hardest to resolve. Once the customer was satisfied 
that the subject matter areas were covered with an 
appropriate view to naval service orientation and mid- 
shipman requirements, the final draft was prepared and 
submitted. 

At this point in time, it is difficult to imagine 
any algorithm for obtaining closure on the definition 
of course content other than the following. 
1. A draft prepared by a behavioral analyst and subject 
matter expert team after extensive reading and dis- 
course among themselves. The former al«>st as a 
pre- conscious operation, can provide input to the 
educational feasibility and probable structure 
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and final form of the material • The latter, using 
his expert knoKledge of the field, makes professional 
judgements on the appropriateness of each item to 
be covered • 

This draft is reviewed by a higher level in-house 
staff, whose function it is to insure overall con- 
tinuity, emphasis, and appropriateness* 
This draft is submitted for review by the customer 
agency, if there is one. As a consequence in this 
review and subsequent discussion, which may be quite 
lengthy yet extremely important, the following 
agreements should be reached* The customer adjudges 
that the content is appropriate and adequate for his 
purposes, and the contractor feels that he has a 
workable set of guidelines for generating a viable 
instructional instrument. Should such an agreement 
never be reached, or is continually subject to change, 
the remaining steps of development are exceedingly 
difficult. 
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This chapter o£ the report has attempted to 
provide guidelines for the firsrt phas-e o£ the model - 
namely, content definition. The other tasks comprising 
the model - terminal objective definition, course and 
media design, materials development, etc. - will be 
treated in successive chapters. 

Those differences of opinion cited and changes 
specified, while honest, were exceedingly costly. 
Those costs would have been higher if not for the develop- 
ment of depth core (see chapter IV, page 208), which 
acted as a kind of safety valve, allowing new material 
to be inserted into the course without fracturing the 
core structure. When the project began, topics such 
as race relations and drug abuse in the military did 
not have the visibility and immediacy that is currently 
attached to them. These and other items were incor- 
porated through depth core units which had more flexi- 
bility than the fifty nine basic core segments. The 
obvious leison is that a similar procedure could be 
employed in any course development effort, or even in an 
on-going revision of an installed course without neces- 
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sitating costly^ change to existing materials. 

Perhaps some day a language will be devised 
that will allow subject matter experts, instructors, 
and educational materials developers to discuss content 
and instructional strategies together - that is, the 
terminology will describe some aspect of content, for 
example, management, in terms that can be meaningful 
to the systematic preparation of that material for 
instructional purposes. Additionally, it would incor- 
porate the means for indicating the presence or absence 
of value or judgment. Thus, all elements for decision- 
making on content definition would be explicit and 
therefore more immediately obvious as points for nego- 
tiation. 
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II I > DEFINING OBJECTIVES - 
STRUCTURING S SEQUENCING RATIONALE 



Abstract 



This section of the report discusses 
the theoretical constructs which in- 
fluenced the preparation of behavioral 
objectives • It offers samples of the 
objectives and their sequencing as 
they actually evolved and the various 
efforts to insure both a rigorous 
structure and an instructionally 
viable sequence • 
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111. DEFINING OBJECTIVES - STRUCTURING § SEQUENCING RATIONALE 

Two general kinds of "sequences" must be constructed 
in the development o£ any course o£ instruction. The first 
is the Content Sequence, which concerns the organization 
of the subject matter; the second is the Procedural 
Sequence, which concerns the organization of th*^ 
instructional presentation. To be sure, there are 
interactions at every stage of course development between 
content and procedure; the content arrangement often 
determines elements of the instructional methods, and an 
analysis of teaching methods may suggest a revised 
ordering of the subject matter. However, because the 
two sequences are constructed more or less independently - 
with the content specified and organized before the 
instructional procedures are designated - it is both 
possible and desirable to discuss them leparately, and 
to indicate at the appropriate places where their 
developments intersect. 

This section is an analysis of the rationale for 

* 

sequencing the content of the Leadership Course. 
Consideration of the content organization was a critical 
factor in the course development. The "sequencing 
rationale, derived from contemporary educational and 
psychological theory, and supported by considerable 
experimental evidence, served as the logical basis for 
the content analysis which the researchers and writers 
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developed. 

General Theoretical Considerations 

First of ally the content of a course may be defined 
as (1) the set of behaviors and capabilities which the 
learner is expected to achieve as a result of the 
instructional experience, and (2) the specific form 
and type of examples, rules, generalizations and 
applications through which the student achieves these 
behaviors and capabilities. As Evans (1968) has pointed 
out, it is not sufficient to specify content solely in 
terms of behavioral objectives, because different students 
may achieve the same objective by different means (i. e., 
by learning different rules and examples; by exposure to 
different training stimuli in different patterns, etc.)* 
The content must be defined in terms of both student- 
performance measures and elements of the task, i. e. , what 
the student will accomplish and by what path' he will 
accomplish it. 

The specification of content was thus a very 
elaborate, detailed process. However, since the learning 
"steps" are derived from the objectives of the course, one 
may describe the content sequence solely in terms of the 
objectives; that is, the content sequence may be defined 
as the planned arrangement of behavioral objectives in 
the order of their expected accomplishment. 
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Briggs (1968) has clearly stated the "educational 

importance o£ the sequencing problem: 

...i£ all the elements o£ skill and 
knowledge ^ ^tudent is to acquire 
during a given course of instruction 
are independent one from another (that 
is, if the 1 earning of one element 
does not facilitate the learning of 
aaother) , then the different elements 
could just as well be taught in any 
arbitrary or random sequence; but if 
the elements are dependent one upon 
another (that is, if the learning of 
one element transfers, thus facilitating 
the learning of another) , then a careful 
sequencing of elements in terms of the 
direction of such transfer should be 
more effective than a random sequence. 

Because the content sequence determined to a large 
extent how a course was organized, it is reasonable to 
believe that the sequence contributed substantially 
to the efforts of the instructional experience in 
producing the desired behavior changes. Such a 
suggestion has been supported by a number o£ recent 
studies (e.g., Gagne and Paradise, 1961; Gagne, Mayer, 
Garstens and Paradise; 1962; Newton and Hickey, 1965). 

To paraphrase Briggs (1968), the value o£ planned 

content sequencing is supported by the expe imental 

finding that learning progresses better under some 

A. 

sequences than under others. There was presently an 
intense r«?search effort to examine those sequencing 
variables which significantly affect the efficiency 
of instructional procedures; it wan WLC's intent to 
^utilize what information was presently available, so 
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that the content sequence vould facilitate the instruction. 

A fundamental concept underlying the development 
of a valid aild useful sequencing rationale was that the 
acquisition of learned behaviors may require prerequisite 
capabilities of other, "subordinate" behaviors. This 
idea, that new learning is built upon prior learning, 
is a basic element of many recent educational theories. 
Although the theories often differ in their description 
of the learning process, in their specific terminology, 
and in their emphasis on particular aspects of the 
instructional situation, they each describe human learning 
as a hierarchical accumulation of acquired behaviors. 
The word "hierarchy" is used, in the broaden, t sense, 
to identify the structural arrangement of a learning 
experience by specifying the relationships between the 
prerequisite and terminal behaviors. A brief discussion 
of three theoretical positions may help to clarify the 
concept of learning hierarchy as related to content 
sequence . 

Bloom (1954), in describing the organization of his 

taxonony of behavioral objectives, wrote: 

Although it is possible to conceive 
of these major classes (of behavioral 
objectives) in several different 
arrangements, the present one appears 
to us to represent something of the 
hierarchical order of the different 
classes of object ives. As we have defined 
them, the objectives in one class' are 
likely to make use of and be built on 
the behaviors found in the preceding 
classes in this list. 
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Gagne has developed a logic of planned content 

sequencing based on both task and performance variable 

In his book, The Conditions of Learning (1965), Gagne 

described eight types of learning, "defined in terms 

of the conditions required to bring them about. These 

learning types are arranged in order from the most 

"simple" (stimulus-response learning) to the most 

complex (application of principles); they are related 

to each other in that for a given unit of content the 

acquisition of the simpler types of behaviors is a 

prerequisite for learning the more complex tasks: 

The most important class of condition 
that distinguishes one form of learn- 
ing from another is its initial state 
in other words, its prerequisites. The 
conditions for chaining, for example, 
require that the individuals have 
previously learned stimulus -response 
connections available to him, so that 
they can be chained. If this condition 
is not met, one finds oreself dealing 
with conditions for establishing these 

prerequisite Ss >R's and thus likely 

to draw incorrect conclusions about 
chaining, itself. This generalization, 
applied to the (eight) varieties of 
learning. . .may be briefly stated as 
follows : 

Problem solving (type 8) 
I which requires as prerequisites: 

Principles (type 7), 

which require as prerequisites: 

Concepts (type 6) , 

which require as prerequisites: 

Multiple discriminations (type 5), 
which require as prerequisites; 

Verbal associations (type 4) or 
other chains (typfjtf); etc. 



Gagne related the hierarchy of learnirig type to 
the rrocess of learning by attempting to show that 
"each variety of learning. . .begins with a different 
state o£ the (student) and ends with a different 
capability for performance." 

In other words, "each of the eight varieties of 
learning conditions .. .establishes a different kind of 
capability in the learner," so that learning may be 
described as the hierarchical achievement of learned 
capabilities. The acquisition of knowledge would 
therefore be defined as the achievement of "an inferred 
capability which makes possible the successful 
performance of a class of tasks that could not be 
performed before the learning was undertaken." (Gagne, 1962) 

Briggs (1968) has echoed Gagne 's conceptualization 
of the learning process as a hierarchical acquisition 
of learned capabilities. He used the term "subordinate 
competencies" to describe those capabilities which must 
be achieved before a particular task can be mastered. 

In a consulting report submitted to the USNA, 
Briggs and Tallmadge (1967) described the necessity of 
studying and analyzing the content and objectives "for 
the purpose of determining the sequence of instruction. 
Included in this analysis would be a detailed specification 
of the 'prerequisite competencies', referring to the 
subordinate skills and knowledges which must be acquired 

«♦ 96 . 



in particular sequences in order to attain the mastery 

o£ a particular behavioral objective." 

The description of learning as a hierarchical 

attainment o£ acquired capabilities has direct 

application to the planned sequencing o£ content, 

since this sequence has already been defined as the 

ordered arrangement of behavior objectives, and since 

these objectives are defined as the expected behaviors 

acquired by the learner as a result of the instructional 

experience. The task of content sequencing thus 

involves analyzing the behavioral objectives to determine 

their dependent and independent relationships, so that 

the achievement of prerequisite competencies enables 

the student to attain succeeding behaviors. The 

rationale for such an analysis has been stated by Gagne (1965) 

Tlie existence of capabilities within the 
learner that build on each other... 
provides the possibility of the planning 
of sequences of instruction. If problem 
solving is to be done... then the... 
principles to be applied to the problem 
must be previously learned; if these 
principles in turn are to be learned, 
one must be sure there has been previous 
acquisition of relevant concepts; and so 
on. Thus it becomes possible to **work 
back" from any given objective of learning 
to determine what the prerequisite learning 
must be; if necessary, all the way back 
to simple verbal associations and chains 
...When such an analysis is made, the 
result is a kind of map of what must be 
learned. 

1 
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It is this "map of what must be learned" which 
represented the basic structure o£ the course. As 
defined by Briggs (1968), course "structure" is 
"the description of dependent and independent relation* 
ships among component competencies, arranged so as to 
imply when seflu^ncing can be random or optional and when 
sequencing must be carefully planned, on the basis that 
transfer will be optional in order to build up from 
simple skills to more complex ones." 

Briggs (1968) has developed a classification 

scheme for a number of possible types of structure: 

!• A flat structure . This phrase is employed 
to describe the organization or structure 
of the course when it does not appear to 
matter in what sequence the instruction 
for the various major objectives or 
subobjectives is arranged. For a course 
with a flat structure one could either 
conduct the instruction in a random 
sequence or in any arbitrarily chosen 
order which one prefers. The significance 
of such a structure would be that the 
competencies gained in reaching each 
objective are independent from the 
competencies gained in reaching all other 
objectives. It is conjectured that if 
a course is well analyzed for the purpose 
of deciding upon the sequencing of the 
instruction, a truly flat structure will 
seldom be encountered. When one does 
encounter an apparently flat structure of 
a course, one can question the value of 
the course, or perhaps the objectives 
should be restudied to make sure that they 
are in fact behavioral objectives rather 
than content objectives (see Briggs et al., 
^ 1907). An example might be a foreign 

language course involving written Vocabulary, 
but no sentence translation or speaking. 
Another might be a history course t limited 
to learning dates on which events occurred • 
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2- A vertical structure . This term' is 
applied when there is one fixed best 
sequence, in which Objective A sHduld 
first be taught, the Objective B;and 
then C, and so on. A course having 
such a vertical structure thus contains 
no "lateral transfer" among competencies 
which would otherwise appear at the same 
level in a hierarchical structure. It is 
believed that this type of structure 
would also be encountered infrequently 
f6r a course as a whole, although it may 
be encountered sometimes in planning 
the instruction within a single course 
objective. This type of structure then 
has only one competence per level. 

3. The hierarchical structure . This kind 

of structuring is represented by a pyramid 
shaped arrangement of the objective of the 
course in which the objective at the top 
of the pyramid is a global, total course 
objective, and the subordinate objectives 
are arranged in layers. A hierarchical 
structure implies that all of the 
competencies within a layer should be 
taught before instruction for the next 
layer is begun (because vertical transfer 
is expected), although there may be options 
ill the sequencing of the instruction within 
a flayer (if lateral transfer is not expected) . 
A hierarchical structure is a frequently- 
reported structure for carefully analyzed 
learning objectives or tasks. 

4* Mi^eA structure. This would be illustrated 
by a course in which two or more major parts 
can be taught in random order, but where 
hierarchies may exist within the parts. 
This is called "parallel learning" by Ausubel 
and Youssef (1963) . A special case might 
exist as for foreign language, where vocabulary 
and sentence structure are both involved. 
Just how vocabulary and sentence s'tructure 
are time phased would permit at least these 
options, desirable or not: (a) teach all 
the vocabulary, then sentence structure; 
(b) teach some vocabulary and some sentence 
structure, using familiar words only in the 
latter; (c) same as b, but introduce new 
wotds in the course of instruction on sentence 
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structure. There may be instances o£ 
the reverse situation^ where sequence 
is important among major objectives but 
not in the learning o£ the individual 
objective. 

The special case o£ a flat structure 
requiring spiral sequencing o£ instruction. 
This type o£ structure is sometimes 
encountered when the major objective of a 
course is that the student learn to solve 
complex problems by analyzing each of 
several major components in a problem* 
An example of this has been encountered in 
discussions with subject-matter experts 
on learning to analyze foreign policy 
problems. Subject-matter experts in this 
area indicate that in making each foreign 
policy decision in such a way as to appraise 
each factor individually and also to arrive 
at the best trade-off decision as relating 
to each of the 12 factors. In arranging 
instruction in this case one might proceed 
first by teaching some individual concepts 
and principles needed later in analyzing 
each of the 12 factors to be considered in 
making a foreign policy decision; this 
constitutes the first '^spiral.** These 12 
introductory sequences can be taught in 
any order. Next, a problem would-be 
presented to the student in whichi the 
analysis is given to him for 11 factors, 
and he solves the problem by analyzing the 
12th factor and arriving at a decision based 
on use of the analyses provided for the 
other 11 (second spiral) . In the next 
problem given, two factors would essentially 
be left blank, requiring the student to 
consider the supplied information on 10 
factors in terms of his own analysis for the 
other two factors. By spiral ing the 
instruction the programmer or experimenter 
continues to supply an increasingly small 
part of the total solution, while the learner 
develops the competency to solve an 
increasingly large part of the problem. 
After a dozen such cycles or spirals he 
has acquired the capability to analyze a new 
problem by analyzing all 12 of its components, 
and making the decision for action. While 
there might conceivably be a bett^er way to 
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to approach such instruction, the spiral 
sequencing at least represents one logical 
approach for this type of problem. This 
type of learning structure has been 
discussed by Glaser (1962) and by Bruner 
(1963), who refers to this sequencing 
procedure as "revisiting,*' or learning 
which "turns back on itself." In the 
"RULEG" system of programing discussed by 
Glaser, a wider variety of examples or 
finer discriminations may be accomplished 
by the latter spirals. 



Although Briggs has outlined a number of possible 
F structural arrangements of behavioral objectives, both 

he and Gagne have hypothesized that the analysis of 
most courses of instruction would reveal a hierarchical 
7 structure, "represented by a pyramid- shaped arrangement 

of the objectives. in which the objective at the top 
I of the course is a global, total objective, and the 

_ subordinate objectives are arranged in layers. (Briggs, 1968) 

Gagne (1965) h?s suggested that "the subjects of school 
]' instruction possess hierarchical organizations with 

respect to required types of learning. Each can be 
\ analyzed to reveal prerequisite learnings that grow 

progressively simpler as one works downward from 

^- principles to Ss connections. 

r And Briggs (1968) has cited a number of studies 



which have: 



...supported the view that many courses or 
portions of them, when analyzed carefully, 
display an inferred hiearchical structure 
whose validity can be supported by comparing 
sequences of instruction designed to follow 
the hierarchy with some other strategy of 
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sequencing. Several experiments provided 
various kinds of evidence of the validity 
of the hierarchy, because when learning 
was sequenced accordingly, learning 
progressed better than under other 
sequencing procedures. 

"In a general sense/' Briggs (1968) wrote, "an 
analysis of the behavioral objectives for any course of 
instruction should be the starting poinc. A study of 
the extent of apparent interdependence or independence 
among such objectives should suggest the latitude one 
might have in blocking out the molar units of instruction 
as to their place in the course, and an effort to determine 
the internal structure of each objective in terms of 
structures which appear to be vertical, flat^ or 
hierarchical in nature should provide a starting point 
for considering the more molecular structure of the course." 

Briggs* suggestion, that the more general and 
comprehensive objectives should first be identified and 
sequenced, has been extended by Stolurow and Brennan (1968). 
According to these authors, statements of behavioral 
objectives can be made at two basic levels. The first 
level is the macro-level of objective, which roughly 
corresponds to behaviors brought about in students as 
the result of -studying relatively large areas of the 
content. The second level is the micro -level of objective, 
which corresponds to behaviors brought about in students 
as a result of studying relatively specific elements of 
the content. 

1C2 ' 
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Macro Objectives , 

Stolurow and Brennan (1968) define macro -objective 

as the overall, large-scale instructional intention of 

the objective or test item. Macro -content refers to the 

general section of the course to which the objective or 

test item is aimed. In this case, macro-content refers 

to the clustering of content into meaningful course 

Chapters (or Parts). For example, a Part concerned with 

communication should have as the basic objective the 

development of skills in communication as evidenced by 

the ability to appropriately apply principles of 

communication in new problemsolving situations. Macro - 

objectives will be stated as expected changes in student 

behavior which occur as the result of interaction with 

Parts or macro -content. They will not necessarily 

include the specific activity the student must perform 

to demonstrate his mastery of the objective. r 
' t 
Micro-objectives, on the other hand, will be stated 

as terminal behaviors the student will engage in, and 

will immediately imply his degree of mastery of the 

objective. These objectives will contain a verb ("virile 

verb") which specifies the exact performance expected 

of the student in relation to the specific content 

measured. 

Stolurow and Brennan (1968) define micro-objective 
as a statement of something the learner will "do". The 
rationale for stating the objective in this way is that 

m 
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the learner* s behavior can be observed and measured, 
i.e., tested, (Mager, 1962; Loree, 1965; Gagne, 1965). 
Micro-content is defined as the very specific area 
of knowledge with which the test item or objective is 
concerned. It is the specific object of the **do" 
statement. Stolurow and Brennan note a major distinction 
between micro- and macro -content in that micro -content 
is always found in a typical statement of an educational 
objective whereas macro -content is often merely implied. 
The value of stating the behavioral objectives at both 
levels is that the macro -objective ma) be used to 
describe the general course structure, and the micro- 
objectives may be sequenced independently within 
macro-content units. 

As already stated, the specification of behavioral 
objectives was a preliminary task in the development of 
the content isequence (primarily because the sequence 
may be defined as an ordered arrangement of the objectives). 

WLC's next undertaking, therefore, was to categorize 
the specified material into individual topic units, each 
defined (in behavioral terms) as a macro-objective. It 
should be mentioned that there were a number of advantages 
in identifying these macro-objectives, besides those 
discussed earlier in this paper: 

a. It was possible to identify the intent of the 

course in a few, rather than many, general 

1 



statements . 




b. The macro-objectives lend direction to the 
sequencing o£ major aspects o£ course 
content and to the sequencing o£ instructional 
procedures. 

c. The macro-objectives can themselves be defined 
by a set o£ micro-objectives. This facilitates 
the efficient classification of content 
material within topic units ^ and also aids in 
planning the sequence of the units. 

d. The macro-objectives can be tested to determine 
the effectiveness of the instructional system. 

e. Tests, based on macro-objectives, will also 
give some indication of long-term retention 
of the principles of leadership. 

After analyzing the content provided, WLC, in 
conjunction with the Naval Academy, categorized the 
content into twelve topic units (or domains) called 
Parts (or Chapters). Each of these twelve Parts was 
represented by a macro -objective. 

These macro-objectives were then sequenced in the 
manner suggested by the theories which have already been 
reviewed. The basic inferred course structure, as 
generated by the sequencing of macro-objectives, was 
found to be a hierarchical arrangement of the Parts. 
The diagram on the next page illustrates this hierar- 
chical structure « 
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^ Hierarchical structure of the topic units (Parts) of the Naval Leadership Coarse 
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It can be seen that the course was organized into 
four (horizontal) levels, each level containing one or 
more topic units. The achievement of the macro -objectives 
at each of the levels was prerequisite to the acquisition 
of knowledge in the next higher level. Mastery of the 
principles of military management, for example, was 
facilitated by the knowledge of the principles of 
individual behavior, group dynamics, and communication. 

The micro -objectives (or "terminal objectives") 
identify the specific learned behaviors which the student 
was expected to accomplish as a result of the instruction. 
The micro-objective closely fits Mager's (1962) definition 
of a behavioral objective as an "intent communicated by 
a statement describing a proposed change in a learner 
a statement of what the learner is to be like when he 
has successfully completed a learning experience," 

It is important to recognize the critical 
relationships between macro- and micro-objectives. A 
macro -objective, although by itself a rather general 
description of achievement, is exactly defined by the 
set of micro-objectives comprising the topic unit. The 
macro -objective may then be considered as a "summary 
statement" of a number of content-related micro-objectives. 
An example of each type of objective, in its relation 
to each other and to content, might be: 
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MACRO-CONTENT (PART) : Communication 
MACRO-OBJECTIVE: the student will be able to 

demonstrate the ability to 
communicate effectively. 
MICRO-CONTENT (SEGMENT): Second principle 

of communication. 
MICRO-OBJECTIVE (TERMINAL OBJECTIVE) : Given 
a list of hypothetical situations, the 
student will demonstrate his understanding 
of the second rule of communication by 
selecting the appropriate situation in which 
it would apply. 
Micro Objectives 

The micro -content and terminal objectives were 
derived from the content. Since this content was 
largely a description of the principles (rules) of 
leadership, the terminal objectives are statements of 
the students* expected in-depth acquisition of these 
principles; the objectives are emphasizing three major 
types of capabilities: 

a. The student will demonstrate his knowledge of 
the rule by being able to discriminate a 
statement of the rule from a number of 
alternatives. 

b. The student will demonstrate his understanding 
of the rule by being able to generalize the 
rule in a given situation. 
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c. The student will demonstrate his ability 
t apply the rule by using it in problem 
solving situations. 
Some advantages of specifying the terminal 
objectives in this way, in addition to those already 
mentioned, was: 

a. The micro-objectives required the student 
to "do" something which can be observed and 
measured. 

b. The micro-objectives gave direction to the 
sequencing of the segment content and 
sequencing of presentation variations within 
segment instruction. 

c. The micro -objectives were tested by a number 
of test items which involve discrimination 
and generalization as well as problem-solving 
abilities • 

d. Tests of the micro-objectives provided valid 
and reliable estimates of the effectiveness 
of segment presentation. 

e. Tests of the micro-objectives made it possible 
to pin-point the student* s specific areas of 
difficulty in learning materials. 
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An individual terminal objective (T.O.) only 
partially defines a macro -objective, and therefore, is 
by comparison a relatively small unit of behavior. 
However, by itself a T.O. represents a sizable 
achievement, requiring a number of prerequisite 
competencies for its accomplishment. Since these 
prerequisite competencies "enabled" the student to 
achieve the final terminal behavior, they are called, 
appropriately, enabling objectives (E.O.) . 

The analysis of each terminal objective - which 
entails the specification and sequencing of its 
enabling objectives - was one of the most difficult, 
most elaborate, and most important tasks in the 
organization of the Leadership Course. It was in this 
task that the theory which we have been discussing finds 
its most effective application, because the sequence 
of the instructional procedures was derived in large 
measure from the sequence of enabling objectives. 

Suggestions from six different theories contributed 
to the rationale for the specification and sequencing 
of enabling objectives: 

1. Gagne (1965) has stated that complex, learned 
behaviors are built up from more simple 
behaviors. Since most of the terminal 
objectives are at the Type 7 (Principles) 
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and, Type 8 (Problem Solving) levels, one 
facet o£ the task is to specify the enabling 
behaviors in terms of Gagne's learning types. 
The analyses completed thus far have indicated 
that the enabling objectives were arranged in 
a hierarchy within each terminal objective. 
Bloom ^s (1965) taxonomy has also been used in 
the analysis of the terminal objectives. Bloom's 
major categories of educational objectives 
(knowledge, intellectual abilities, and skills) 
have served as the basis upon which the T.O.'s 
are measured; the student will demonstrate 
his mastery of each T.O. by being able to 
discriminate the rule, generalize from the 
rule, and solve problems by applying the rule. 
Each of these requirements indicated the 
content and form of many of the enabling 
objectives. 

Briggs ' (1968) discussions of course structure 
based on hierarchies of competence have also 
contributed to the construction of the E.O. 
sequences . 

The RULEG system, proposed by Evans, Homme, 
and Glaser (1962), contributed procedures for 
analyzing the T.O.'s and E.O.'s in terms of 
their rules and examples. This method also 
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considered the hierarchical nature of 
learning based on the type o£ discrimination 
and generalization required. 
Gilbert's (1962) competition-facilitation 

analysis was used extensively in the designation 

I 

of the enabling objectives. This procedure 
dealt with other elements of transfer besides 
hierarchical facilitation. It was essentially 
an analysis of behavior domains in which 
discrete components of subject matter are 
identified and grouped. Gilbert's methods 
suggested which behaviors should be taught 
together, which should be taught separately, 
and the general ordering of the instruction. 
Tosti and Ball's (1969) media selection model 
considered the importance of planning the 
sequence of instruction. These authors 
included sequence within the "distribution" 
dimension of presentation. This dimension 
referred to "the ordering, grouping, and 
sequencing of behavioral items and the 
temporal spacing between such items • 
Distribution included such items as the 
distribution of practice, the frequency of 
review and the hierarchy of content presentation.' 
Describing the concept of behavioral hierarchies, 

V t 

Tosti and Ball (1969) write: 

112 



1 
I 



It is known that behavici^ which can 
be built onto already existing behaviors 
is generally facilitated in its 
acquisition, retention, and generalization. 
This is one of the primary differences 
between the application of modern 
behavioral theory and the application 
of behavioral theory in the twenties. 
Thus, if a new stimulus-response (S-R) 
pair is going to be installed, it would 
be most advantageous to have certain 
other S-R pairs previously established. 
Of course, such antecedent behaviors 
may have been established before the 
instructional setting, or the distribution 
decision to create and establish such 
behaviors within this instructional 
setting might be made before the 
establishment of the new behavior. 
There are several classes of such 
behaviors which must be considered. 
These include stimulus discrimination, 
response differentiation, existence o£ 
mediators such as understanding, 
analogy, or mnemonic and those 
response sets concerned with 
observing and covert responding 
sequence. 

The specification and sequencing of the terminal 
and enabling objectives presently completed was based 
on the theoretical positions we have just described. 
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Some Prottlems and Their Resolu t ions 



At a more detailed level of specificity, the 
following examples and charts may illuminate the problem 
of defining objectives in a soft science such as leader- 
ship, psychology, and management, structuring and sequenc- 
ing them. 

The first and most pbvious problem is the potential 
conflict that comes in an hierarchical arrangement of 
content topics and the hierarchical analysis of objectives. 
The two are not necessarily coincident or congruent. Still 
another problem, but not discussed here, is the necessity 
of congruence between content topics and objectives on 
the one hand and the actual instructional material as it 
is developed by the writer. For example, a content topic 
might be listed first in the outline because of its 
importance to the subject matter expert and the objective 
derived from that content is likewise placed at the initial 
step. The writer, on the other hand, employing an 
inductive approach to the materials, may relegate that 
content and objective to some other area of the instructional 
program. 

The problem of fit between content topics and 
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objectives can be a persistent one. In this project, 
however, the objectives were derived from the content 
topics. While some may argue that task analysis o£ 
requisite behavior and the subsequent sequencing o£ the 
result is the most desirable ray to generate objectives, 
content specification was given priority by the customer, 
who specified that terminal and enabling objectives would 



be derived from major and minor content headings. The 
question raised at this point, is whether or not the 
assumptions used in the ordering of content topics was 
done with sufficient wisdom such that an hierarchical 
structure of objectives would be successful. The following 
series of extracts pertaining to Part 7 indicates that the 
hierarchical analysis of topics as initially defined by 
both subject matter experts and writers, has a remarkable 
congruence to an analysis o£ the objectives as they are 
actually sequenced. In addition, they both reflect a 
viable sequence from simple to complex learning. 

Table 3, an extract from the content outline from 
Part 7, Segment S, specifies in some detail both the topics 
to be covered in participative leadership , as well as the 
relationship of the topics. While not a true hierarchical 
arrangement, the extract indicates the relative subordinance 
or dominance of all topics « 




103 



Table 3 

Content Outline 
Part 7 Segment 5 

CExtract) 



J16 

104 



• TABLE 3 ' 
PARTICIPATIVE LEADERSH I P 



A* Introduction (Petrullo and Bass; Filley and House) 
!• Research 

a. Results show that a democratic (participatory) atmos- 
phere produces better morale than an autocratic 
atmosphere; that is, 

Less hostility 

More cooperation 

Greater friendliness 

b. Experimenters have found that leaders who use parti- 
pative leadership are significantly more effective 
than those who use a highly formal structure^ 
(Anderson and Fiedler): 

Made a greater contribution to group's performance. 

J Personality became more salient and more highly 

t relevant to group achievement (Hollander and Julian, 

1969). 

2. Importance 

a. Improvement in group (follower) performance through 
participative leadership: 

Leader should participate rather than merely supervise. 

Leader will encounter less resistance and get higher 
productivity from group and individuals who formerly 
resisted change when they are allowed to participate 
in decisionmaking concerning their work (Hays and 
Thomas, p. 40). 
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b. An effective leader encourages subordinates to offer* 
suggestions and/or constructive criticism. 

3. Practical Implications 

Generalizations from research studies conducted under 
strictly controlled laboratory conditions in industrially 

1r "5 117 



TABLE 3 cont. 



oriented settings were not typical o£ practical shipboard 
situations and should be avoided. 

The situation is an important determiner in leadership 
style. 

B« * Participatory vs. Supervisory 

1 . General 

A leader who uses participative leadership: 

a. Must be attentive to his followers* suggestions and 
reactions (not just lip service). 

b* Retains his authority and responsibility under both 
participatory and supervisory leadership. 

c. Must understand that the degree of participative lead- 
ership depends upon the leader, followers and the 
situation. 

1) Leader: 

Confidence in his petty officers 

Concern for his petty officer's opportunity and 
motivation to demonstrate performance in decision- 
making 

2) Followers: 

Attitudes with respect to assuming responsibilities 

Experience 

Competence 



An effective leader seeks responsibility and develops 
a sense of responsibility among his subordinates. 



3) 



Situation: 
Time available to make decisions 
Type of organization 
2« Advantages 

An officer who practices participative leadership: 
Will probably increase morale within his organization, 
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TI^BLE 3 cent. 



C. 



Helps to inspire enthusiasm among his men, 

Helps develop leadership skills on the part of his subor- 
dinates, 

Strengthens his communications channels with his followers 
through improved feedback, 

Through the use of committees, when time permits, will 
find that the skills and knowledge of his men are probably 
greater than he imagined. 

Will find the men likely to respond with extra effort in 
emergency situations. 

a. An effective leader, upon reassignment, can answer 
* affirmatively : "Are the men I have just' left, better 
men for my having served with them?" 

3. Barriers to Participative Style 

a. Lack of time 

b. Decisions contrary to interest of followers 

c. Followers unwilling or not motivated to follow 

d* Leader's fear of failing because he might lose control 
of the group and the situation or, because of past 
failures. 

Summary 

1. Democratic, participatory leadership: 

a. Produces better morale, less hostility, more cooper* 
ation and greater friendliness. 

bt Produces leaders who are significantly more effective 
and make a greater contribution to group performance. 

c. Must be tempered by the situation. The situation is 
an important determiner in leadership style. 
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A quick reading o£ Table 3, in conjunction 
with the extract o£ course objectives [Table 4 ], indicates 
both an ordering of the terminal and enabling objectives 
as well as their relationship to the content outline for 
that segment. It is noted that each terminal objective 
has a relationship to major content headings in the 
outline. 

An independent analysis of terminal objectives to 
determine ascending difficulty of cognitive requirements 
or skills is shown in Table 5. The cognitive behavior 
requirements, listed on the right side of the table, 
indicate a successive degree of abstract thought as the 
student would move through the various tasks specified by 
the terminal objectives. 

Content maps are discussed in more detail elsewhere, 
I but the content map supplied in Figure 7 was a further 

attempt to analyze this segment to determine: first, if 
a hierarchical arrangement and a logical ordering did 
exist in content definition, and secondly, if the objectives 
reflected a congruent hierarchy. 
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Table 4 
Objectives 
Part 7 Segment 5 
(Extract) 
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TAiibH " 



Outline 
Reference 



Terminal and Enabling Objectives 



A.l. 



The M will be able to identify, from several 
choices, statements that correctly describe 
the general results of studies done on parti- 
cipative leadership. 



TO 1 



A.2. 



The M will be able to select, from 
several choices, the statement which 
correctly identifies the effect 
participative leadership may have on 
a group's resistance to change. 



EO-1 



A. 3. 



The M will be able to identify, from 
several choices, the dangers involved 
in making generalizations from labor- 
atory studies to shipboard life. 



EO-2 



B. 



Gi'/en examples of leaders interacting with 
groups and using various leadership styles, 
the M will be able to identify the situa- 
tion in which the leader uses participative 
leadership. 



TO 2 



B.l.a. 



The M will be able to select, from 
several choices, the statement which 
correctly indicates how the leader 
must treat the suggestions of a subor- 
dinate in order to ensure successful 
participation. 



EO-1 



B.l.b, 



Given a description of a situation in 
which the leader is using participative 
leadership with his group, the M will be 
able to identify whether or no'c the 
leader's authority is weakened. 



EO-2 



B.i;b, 



Given descriptions of leaders utilizing 
participative leadership, the M will be 
able to identify whether or' not the 
leader is relieving himself of his 
responsibility. 



EO-3 
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Outline 
Reference 



Terminal and Enabling Objectives 



Given a description of a group, the M will 
be able to select, from several choices, the 
action which the leader should take regarding 
participation. 



TO 3 



B.l.Cc 



Given several descriptions of the job 
competence of a group in relation to 
the job competence of the leader, the 
M will be able to select those descrip* 
tions which indicate that the leader 
should use participation. 



EO-1 



B.2. 



B 



.2* 



The M will be able to identify, from several TO 4 
choices, statements that correctly describe 
the effects of participative leadership when 
used by military leaders. 



Given a list of possible advantages EO-1 
accruing from the use of participation 
(both valid and nonvalid), the M will 
be able to select those which are valid. 



1 



B.3. 



Given a description of a situation in which 
a leader has the option of permitting parti- 
cipation of subordinates or not, the M .will 
be able to select, from several choices, the 
correct action for the leader to take (based 
on evaluation of the barriers inherent in 
the situation) 



TO 5 



B« 3 • a • 



Given several descriptions of situations 
in which participation might be used, 
the M will be able to identify those in 
whicK participation could not be used 
because of a time factor. 



EO-1 
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TABLE 4 cont. 



I 
I 



Outline 
Reference 



Terminal and Enabling Objectives 



B 1 3«b* 



The M will be able to select, from TO S 

several choices, the statement which EO-2 
correctly describes a harrier to 
participation resulting from a 
subordinate . 



The M will be able to select, from EO-3 

several choices, the statement which 

correctly indicates how fear of 

failure on the part of the leader 

might prohibit the use of participation. 
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Analysis o£ Terminal Objectives 
Ascending Difficulty of Cognitive 
Skills 
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PART SEVEN, SEGMENT 5 ~ PARTICIPATIVE LEADERSHIP 
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Terminal Objectives 



TO>S. Given a description of a situation 

in which a leader has the option of— 
permitting pai'ticipatlon of sub- 
ordinates or not, the M will be able 
to select, from several choices, 
the correct action for the leader to 
take (based on evaluation of the 
barriers inherent in the situation). 



TO-4. The M will be able to identify, from 

several choices, statements that 

correctly describe the effects of 



participative leadership when used 
by military leaders. 



TO-3. Given a description of a group, the 
M will be able to select, from 



several choices, the action which 
the leader should take regarding 
participation. 



TO-2. Given examples of leaders interacting 
with groups and using various leader-- 
ship styles, the M will be able to 
identify the situation in which the 
leader uses participative leadership. 



TO-1. The M will be able to identify, from 
several choices, statements that — 
correctly describe the general 



results of studies done on participa- 
tive leadership. 
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Cognitive Behavior Heguircments 



Differentiation of multiple causes/ 
-multiple effects 



Projective cause-effect understandings 



Prescriptive selection from a group - 
. applying principles of subject matter 



Simple discriminations to isolate 
-concrete applications of the 
definitions 



Understanding broad generalizations 



1?6 
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' There is some difficulty in specifying the rela- 

j tionship between course objectives and the classifications 

L 

as established by Bloom and others. A detailed exposition 
I of all considerations is detailed in TR 6.12. For purposes 

of this report^ one can summarize two conditions. It was 
specified early in the project that all terminal objectives 
I and subsequent test items related to them would be problem 

solving^ that is: the midshipman would be given a situation 
J or a description typifying a vrue-to-life situation. • 

He would then be required to select the best course of 
action from his knowledge of previously delineated prin- 
ciples. To the degree that this was possible^ terminal 
objectives and the test items that proceeded from them 
were thus constructed. Problem solving^ however, is a 
higher order abstraction as defined by most taxonomies 
and the effort to make "problem solving** oriented conditions 
for all terminal objectives was not always successful. 
However, it can be summarized here that according to the 
letter, if not always the intent, 90% of the terminal 
objectives are at the Type 7 [Principle] and Type 8 
[Problem Solving] levels of Gagne's learning types. Almost 
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all (97%) enabling objectives had equivalence to Bloom's 
categories one through four or other lower order learning 
classifications. In addition, they are sequenced in 
levels of competency within each cluster. 

Previous discussion centered on the relative 
importance of several theoretical constructs used as 
actual guidelines, preparing, structuring, and sequencing 
the objectives. The question naturally arises as to the 
exact relationship between those theoretical constructs 
and the end product, also the resolution of conditions 
where conflicting theories suggested alternative paths. 
These problems were somewhat obviated by the specification 
from the customer that all objectives were to follow 
topic headings. Secondly, most theoretical models tie 
the problem of content specification to the writing of 
objectives. Since the definition of content came before 
the definition of objectives, and since the customer 
specified that objectives would follow content, much of 
the theoretical consideration for content structuring via 
objectives was moot. 

The practical application of the theoretical constructs 
had its greatest impact in articulating the objectives in 
such a manner as to provide rigorous guidelines for the 
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writers of instructional materials and the test items. In 
conjunction with this, was the problem of translating 
substantive content as determined by topic headings into 
instructional ly viable sequences. Also, given the 
constraints of the existing content structure, there was 
the concurrent problem of retrofitting all components of 
a well-defined objective around previously specified 
content. Where the structure naturally lent itself to a 
smooth ordering of competencies, the writing of objectives 
presented no difficulty. Unfortunately, this was not 
always possible and some objectives had either a forced 
fit, or were delinquent in satisfying the requirement of 
a well- constructed series of objectives. 

On the whole, the potentially deleterious effects 
from these conditions did not materialize in the course 
of materials development. For while some objectives 
initially had some structuring and sequencing problems, 
the continuous revision process over the course of the 

project remedied any deficiency. The conclusion here. 

1 

as with other elements of the developmental process, is 
that empirical data will provide all necessary guidelines 
for revision of course components. 

Thus the technical problems of constructing objectives 
can be separated from theoretical considerations for 
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sequencing. That is, given content statements, the 
writer of objectives can specify precise details for the 
action, conditions, criteria, etc. The rationale for 
micro -sequencing of objectives presents other problems. 
The macro-sequencing by part, segment, and major topics 
has been treated previously. Micro-sequencing attends to 
the ordering of enabling objectives under terminal 
objectives, and in some cases in ordering of two or more 
terminal objective clusters. Here theoretical schools 
offer no single solution. Hence, an eclectic rationale 
evolved. Most hierarchies of learning levels are scalar 
in nature and peg certain competencies at one level only. 
It would seem in reality, that the learner proceeds through 
a series of steps that might be analogous to nested DO 
loops in a fortran program. The learner, even at rudimentary 
levels, does some analysis, synthesis, and evaluation as 
he deals with gross knowledge. Each terminal and enabling 
objective cluster, as ultimately defined through many 
revisions of the course, suggest that learning competencies 
proceed in a cyclical fashion through all levels of learning 
hierarchies. Bloom's six classes of knowledge, comprehen- 
sion, application, analysis, synthesis, and evaluation are 
repeated again and again in all or in part, but in actual 
fact, some elements of analysis and synthesis are ordered 
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before application* 

In sum, theories were not used to establish content, 
and when bounced against the realities of an entire course 
structure, they fused into a workable mechanism for 
for micro- sequencing terminal and enabling objective 
clusters. The conclusion suggested is that scalar 
ordering of competencies in a locked step pattern is 
untenable for course construction. Competencies, as well 
as content, can have a cyclical pattern in an unending 
sequence of learning experiences. 
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An illustration o£ this process is seen by examining 



the terminal and enabling objective clusters in Table 
Terminal objective 1 asks that the student select a 
description o£ a^ higher order interrelationship between 
elements. It is supported by two enabling objectives - 
EO^l, on a simpler level, asks for a selection o£ an 
appropriate definition; EO 2 basically asks for a demon- 
stration of understanding of one element. Thus, the 
student moves from simple definition to simple relationship 
of parts terminating in an understanding of larger rela- 
tionships. Terminal objective 2 has its own cluster. 
Ultimately, he should be able to choose the best description 
of an abstract concept. The enabling objectives, leading 
the student to that conclusion, successively require proper 
identification of examples of individual components of the 
total concept and some lower level analysis. 

It should likewise be noted that there is a progression 
from Terminal Objective 1 to Terminal Objective 2. The 
former establishes the overall framework necessary for 
understanding of a specific variation within that framework. 
Neat classification by any one taxonomy, therefore, is 
difficult. The overriding conclusion suggests a cyclical 
hierarchy of learning competencies. 
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7ABLE 6 



Outline 
Reference 


Terminal and Enabling Objectives 




A. 


Given several statements about leadership 
behavior and leadership style, the M will 
be able to select the statement whicli 
correctly describes the interrelationship 
between them and the situation (based on 
tlrif^ concent of leadership stvlel . 


TO 1 


A.l .a .b* 


The M will be able to select, from 
several choices, the statement which 
correctly defines leadership style in 
relation to a leader's responsibilities. 


EO-1 


A.l.c. 


The M will be able to select, from 
several choices, the statement which 
correctly describes the relationship 
of the need structure of the leader to 
his changing behavior. 


cO- C 




Given a description of an officer leading 
a group, the M will be able to select from 
several choices, the correct classification 
of the leader's style according to Fiedler's 
concept . 


TO 2 




Given several descriptions of an officer 
leading a group, the M will be able to 
select the example which illustrates 
relationship-oriented style . 


EO-1 


B.l.b. 


Given several descriptions of an officer 
leading a group, the M will be able to 
select the example whTch represents 
task-oriented style. 


EO-2 


B.l.c. 


The M will be able to identify, from 
^^several choices, the paragraph which 
correctly compares relationship-oriented 
and task-oriented styles. 


EO-3 
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One of the tools for analysis of the objectives for 
this project were content maps - a structured, graphic 
arrangement of the objectives by the content contained 
J within each. While mentioned elsewhere, they're discussed 

here with the further purpose of examining not only the 
validity of the hierarchical structure, but to assess 
their fundamental usefulness as developmental tools* They 
were developed in this form (Figure 8) after the fact; 
I that is, they were not used as development tools, but for 

post hoc analysis. During the development stages, 
hierarchies were contructed informally by the analyst 
prior to structuring and sequencing, but no formal record 
at this level of detail was prepared. 
I During the project, the content maps gain some 

additional consideration since they were incorporated into 
I the instructional material as pre-organizing devices for 

I the students. The evaluation of their effectiveness 

showed no statistically significant benefit, although it was 
I generally agreed that such one page segment charts might 

be useful for both preview and review. Additionally, some 

I 

L controversy arose over the conventions used by the maps 

|- in structuring the content of the segment. Out of this 

i 

~' disagreement, the maps as typically structured in 
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FIGURE 8 

Content Map 
CSegment 4.3) 

A Graphic Representation o£ 
Topic Relationships Within 
a Segment. 
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FIGURE 9-1 
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Figures 9-1, 9-2, and 9-3, were abandoned and removed 
from the instructional material and were replaced by a 
simpler format for inclusion in tlie Student Guide CFigures 
10-1, 10-2) • The central issue for discussion here is 
the relative value of the maps as developmental tools* 
Whether formally constructed before, during, or after 
the writing of objectives, they yield a product which 
has several advantages. The first advantage is that 
they summarize in a simple fashion complex content for 
quick assimilation. A new staff member or a non--project 
reviewer can quickly comprehend content details. Secondly, 
the structure, in its graphic form by whatever convention 
seems appropriate, establishes the fact that the materials 
are indeed logically ordered. Finally, they serve both 
as a test and a tool of congruence between content, 
objectives, and instructional materials. 



129 J41 



FIGURE 10-1 



I 



•» 9 O 1 M 

•1 M g- o 1 

•t O e O 

S >1 9-0 C 



» n o 

1 ^ ft o s. 

*^ t> VI »<|tTS 

« (A O W 

) ««3 tf> W »i f» 3 

t O O ■« M 

» V» » 

t r« » «< f| H> 

•I ** »l »l " 

fCO 7U O C 
9 <9 9 r O 



O M f» 7 «^ 

a (A » r 

t« M to < > 



1 O 3 »* 1 

r c c •*» y o 



O li I* 
C S'O 

n n 



•i o 



I 1 <• c ■ 
O *» t» f» o o 



f» art 9 

f» f n n 
f» w o A w 

o c 3 

O 1 fk 



•i 1 M C 



C 

» so 

3 « » 1 



515 



1 a 



o »• r. « 
} 1 w b 3 

1 »- 3 •- »t 

- »l C 1 

V VI ^ O O 

•O C 
IT 3" 

« O C *l » 

^ 9 ar o 

3 1 f| 1 1 

■» O » O V 

- 3 O C 

- 1 tJ (J 3 



W ^ C* ti •» o • 



r<''A3<^>1C>'>1l 

3 M u-a 3 *^ t 

M O M < .--a y 

f| »^ 3 »#• O r« b 

1 CM 1 1 3 ( 

n 1 •< few J 



M t> a 

3 ' 

O 



3 

n 
9 O 



I O VI 
* 3 1 



o n o 
►-»o c 



3 O 
U C 



•i f* or If n M r» 

»* •< O f| o 
3 •*! 9 

X «1 O - 9* 

♦♦3 1 9 » 



I (A V> 

t n n 

ft* M 1 

r n TO 

10 11 



o 0*0 
O 



1 VI < 
t« O H 1 

3 O 3 
•n 9 1 

on VI 

1 1 b «t 

1 ft 

1 •» o o 

» O 

r 

n « H < 

t*X Ul I 



O li li 

<^ c c n 
•trj w 

M 4 O 

M tl M 
•«\ r» O 

• ar "^li 

n 1 

*» 3 1 

o B. n 

3 ar«-> 

O 3 »• 3 

at* 

c 

H TTU n 
3 1 ^ O 
B. 9 

H O H»H» 

< 1 O O 
B.t> iMti 
C 3 O 

•i M. r M 

M>M -'X 

St a 



> 1 O r« 
• C.3 9- 
VI li 1 



1 3 O 
k. 1 O 3 



— 1 o 
M e 
y 1*0 
i 

1 HtS 

9 1 O 



25? 
1 ^ 

3 1 

^ n 



o c 

9*0 1 

O <^ 

» M 

*«• O 

V •«> 

O - 

n o c 

o ♦*» 

M » 

^ 3 1 
•i &t 

m 1 

n 3 



t3«-n o 
*» ** e 1 

VI «• 3 

n M M n 

(r»'C^*-'««>9M: 
^ 3 a CT-y** or IF 

ft •O^11««01fl 

n ar 1 ar - 

a H s'-f'o 1 ar r 
< •'•^lo'oni^' 
in 'A 3 nor« 

0 i«o a 1 M M-ti c 

•O r« t» C 1 I 

1 ft 3*9 1 M ft b 
111W10 9 = 
% 9 «• S C 
« •» V - ft 

X *-* M V 
O ^ M on (» - 
9 ft *< C ^ O S 

(^•3 ar o o c 
n*-i*>«B^ 
o o 4* n o 1 

C 9 9- ft ^ C *■ 
O w ft w» *■ 



3» 

n O 



or o ITJ o o 

1 *-i c n 9 

im V, o 

C t C - 3 ft 
W ft »-»*'0 3 

«i ^ ft M ^ a, 
^ o 
o *" 3 «t a. 
M 9 o ar •> 
o- 9 n i v^i 
ft • •*« 

ft o ^9 9 1 

»* o n #♦ ^ 
9 o ar I* 
ft n 1 1 9 

9*^ n «i »- 

M> 1 ft 

ft ft OM 

9 M 9 e *> 



) 1 9 

I ft o 



s: S' i? 

1 M M 

9 r n 

n #♦ <A ^ o 1 ft " 

9- H M *" 9 9 • 

ft cr ^ ckc 

*<••< C 1M 1 ' 

9 n 1 M 

• M r« O M 

y C 9-C 7 

•*|1 1 1«0 1 

0 1 

1 M 1119 

ft"< S ♦** 1 
•O 9 ^ 

0 1 1 cr 1 c 

1 WW n 1 n V 

•1 f4 •I M 

•< O O W ft M O 



1 > 

- H 

i 

6 1 9 
M M> O 1 

»1 ^ 
1 ** 
c 

<A O O O 
O VI 9 

• I 
C H-O ft 

f4 1 M 

^ O 9 
O 9 

#♦ n 



I 1 Hf* 

» ft arft H*o 
I ft e 

r 1 < f 1 

» ^ »*•«« H'O 
t VI o 1 f* f 



O 1 " 
9 9'*> 

ft ^ 

ft < n 

2e r 
S 



O WH 

§ 



ar 

Vl5 



ft & 

M H 

c o 
ft ^ 

M 1 

o'l 

9 M 



3^ 



n w 



v 



ERIC 



130 



142 



FIGURE lU-2 
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This chapter has detailed the various conceptual 
and practical complexities that wre involved in the 
definition o£ objectives and their sequencing. For 
purposes o£ the development model, it might be suggested 
that this step o£ the development process is hindered 
by the limitations o£ the state o£ the art in cognitive 
theories o£ learning. Any mistakes that are made in 
assessing the construction, placement, or sequencing 
o£ objectives becomes manifest first when test item 
development begins. The writers of test items can 
attest to the clarity and specificity of objectives 
when they actually attempt to generate items based on 
the prepared objectives. After this test, the value of 
the prepared objectives awaits student performance data. 
Here is the critical test since in the preparation of 
objectives, assumptions were made about the methodology 
both for presentation of content and for eliciting 
desired behavior. The failure of students to perform 
at criterion levels may, at times, be traced to poor 
objective definition and sequencing. 

An interesting final note on objectives is that 
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no one, in th.e course of the project, questioned the 
necessity of objectives.- They were ignored by some, 
but Kere never tHe center of controversy that is some- 
times generated ^en instructors or other educators 
feel their subject province invaded, threatened, or at 
least diminished by the use of behavioral objectives. 
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IV. RATIONALE FOR STRATEGIES. 

MEDIA SELECTION, A ND 
INSTRUCTIONAL $Y3TEMs~gE?lGN 



Abstract 



This section discusses the evolution 
of the instructional systems design 
from a conceptual model through sev- 
eral iterations to yield a final 
course. 
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IV. RATIONALE FOR STRATEGIES, MEDIA SELECTION. AND 
INSTRUCTIONAL SYSTEMS DESIGN 



The General Model 

In 1969, the AV Communication Review carried an 
article in its Spring release that generated much 
critical discussion. "A Behavioral Approach to 
Instruction Design and Media Selection" made the 
bold statement that all media functioned as "cans" 
transporting the instructional message but not signifi- 
cantly altering that message. Such a notion was 
contrary to the McLuhan concept of message modification 
by media which had become an integral part of the mass 
culture during the 1960's. The authors, -^osti and Ball, 
of Westinghouse Learning Corporation, based their paper 
on documentation that had already been submitted as 
part of the project design documentation for the 
OE/USNA Leadership Multimedia project. The paper 
further suggested that media selection should be based 
on variables other than gross media by generic or 
specific type. They suggested that the purposes of 
instruction are better served by attention to those 
presentation variables that are characteristic of the 
media. Thus the ability of a medium to allow questions. 
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to present information in a persistent or transient 
format, to mknage student movement through the instruc- 
tion, were more significant than media per se. 

This position was first suggested in the proposal 
for the project and in the first technical report 
submitted during the design stage. It was the pivotal 
component of the project design since it functioned 
both as the fundamental hypothesis of the research 
design and as guideline for the course development 
model and system. It served both the research activity 
and the course system by offering very discrete and 
manageable instructional variables for experimentation 
to the former, and, to the latter, an eminently logical 
theoretical construct for planning an instructional 
system. It was therefore unfortunate that during the 
life of the project a contention grew out of the notion 
that the project was serving two masters alternatively 
and satisfying neither. Specifically, some considered 
that the research effort testing the validity of the 
Tosti-Ball model, interfered with the proper development 
of various aspects of course development i.e., media 
choice, content definition and the course system. The 
research became a kind of scapegoat. Such a myth was 
unwittingly encouraged when, in the absence of clear 
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empirical evidence to decide differences of opinion, 
» research methodology was offered as a method of 

I resolving the issue. CSee Chapter VIII for furth.er discussion.) 

In reality, if there had been no research effort 
I to test specific hypothesis of the Tosti-Ball model, 

I the constraints that would have derived both by the 

logical construct of a media model and the systems 
I requirement for any rigorous course system development 

would have created some discontent. 
I Although the research data were to indicate that 

I the Tosti-Ball thesis on effectiveness of varied 

presentational forms was not productive for this type 
I of course, its value as method or rationale seems, even 

now, to have great merit as a pioneering focus for 
I considering the many variables that must be thought 

I through for media selection and development work. 

Their thesis started with the complaint that the 
I major fault in instruction design today was the frequent 

I failure to appreciate the distinction between three 

separate design elements: the medium, the presentation, 
I and the method used in preparing the presentation. The 

authors took the position that the medium was no more 
I than the "can" in which the presentation is delivered 

I and that every medium or media mix is superior to every 

* 

I 
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other one for conveying some specific presentation form. 

This was of particular iinportance in the design, 
since the quality of such a system cannot be defined 
in terms of its content or aspects of information 
delivery but only in terms of change in student behaviors. 

Their report assumed that the behavioral approach 
to instructional design has been adequately validated. 
However, the authors modified this approach by separating 
the design of the presentation from the design of the 
operational system (which includes considerations of 
the medium) and have included student repertoire shaping 
as a parallel development. 

Three design possibilities for overcoming media 
limitations were offered: 

1. the mixing of media simultaneously or sequentially 
to overcome the limitations of a single medium 
(derived from presentation design) 

2. the modification of the behavior of the student 
to compensate for the limitations of the medium 
(derived from student shaping design) 

3. employing whatever is available and being 
willing to settle for the resulting attenuation 
of the learner output (null case) 
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A significant part of the behavioral approach 
to instructional design was the engineering of presenta- 
tion. Presentation design was discussed in terms of 
the following five dimensions: 

1. the encoding form for the information 

2. response demand characteristics of the 
presentation 

3. management of presentation 

4. duration of presentation 

5. distribution of presentation 

Their instructional design model stated below 
relied on three informal hypotheses for its strength. 

1. Behavior engineering is a practical science. 

2. Media are best characterized in terms of 
limitation as opposed to advantages. 

3. Presentation form and media-Mx are independent. 
It was proposed that an engineering approach ra\her 

than a psychometric approach be taken in the develop- 
ment of instructional systems. That is, rather than 
measuring students and selecting those who best fit, 
it seemed more advantageous to design a system in which 
necessary presentation is available for anyone. , 
The developmental system, then, should start with 
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specific emphasis on the two important areas: presen- 
tation design and media selection. 

The Engineering^ of Presentation Forms 

Step 1, Determine the nature of the problem by 
interview, observation, or research. Establish general 
goali to solve the problem. 

Step 2. Determine the specific behaviors to be 
established and the entering behavior of the students. 

Step 3. Deduce the presentation factors which 
produce the desired behavioral effect employing established 
evidence in learning. Then analyze or synthesize the 
generalized response sets which may be employed by the 
student in his response to the presentation. 

Step 4. Select media which fit the presentation 
requirements. Media selection must be done in terms of 
eliminating media which limit or otherwise adversely 
affect the presentation design rather than specifying 
advantageous media. Then assemble an operational 
instructional systems package Q^edia mix). 

Step S. Determine strategy for introduction of the 
operational system into the instructional environment. 
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I£ possible, each step imposed as few constraints 
as possible on the preceding step. The final system was 
to be extended to handle all constraints, but the fewer 
constraints imposed, such as prior medium selection, 
etc., the better and leSs expensive will be the resulting 
system. 

It was felt that other media researchers to date had 
not chosen to distinguish a presentation form from the 
media which carry it. The new model required that such 
a separation be made. 

Presentation forms were structured to communicate 
all data (stimulus, response, system control, student 
control) necessary for an efficient student- system inter- 
face. The hypothesis stated repeatedly that a student 
does not learn from the media, he learns from the 
presentation form. 

The authors* background in programmed instruction 
suggested the use of presentation forms in the programmed 
instructional form. Since the typical PI presentation is 

highly structured, the following aspects of the Tosti- 
Ball model were derived: 

1. The stimuli are presented in a verbal, or 
illustrated form. 

2. A demand of a written or selection response 
is made. 
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3* The presentation lasts as long as the student 
desires • 

4. The student must make some response before 
proceeding to the next item* 

After experiments in developing programmed materials 
for several years and using them in the Behavior Systems 
Division of WLC^the need for a separate presentation 
design became clear to the authors* 

A resulting operational system employed PI texts, 
group discussions, progress check tests, and various high 

probability motivation activities, such as chess, as 
the instructional media* The system was so successful 

that almost any medium coulc be thrown in books, 
lectures, exercise sheets --in place of the PI workbooks, 
and the system still worked. After such long devotion 
to the careful analysis of the stimulus and the response 
aspects of presentation, it was found that 
other factors, such as instructional management, could 
compensate for poor design* This forced the designers 
to cease considering PI as being synonymous with the 
workbook medium and to consider PI as a presentation form* 
On the surface, considering the limitations of 
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media may seem to be nothing more than the reverse 
of stating the advantages of media. The difference 
was subtle but important. Two media may fit the 
general requirements of the presentation , but each 
has different limitations. It may be easier to com- 
pensate for the limitations of Medium A than of Medium 
B. Hence Medium A should be selected. Quite often, 
however. Medium B has an aura of advantage surrounding 
it. It is multi-sensory, it is new, it has been 
successfully used in other settings, or it has gotten 
great press reviews, etc. The novice designer may, 
therefore, incorrectly select Medium B because of its 
intrinsic advantages. 

The assertion that media limitations are the key 
to proper media selection does not outlaw at least two 
special cases where media advantages should be used. 
Suppose the situation arises where at least two media 
have almost equivalent limitations for a given presenta- 
tion design. Then, if one medium has the advantages of 
greater student experience or preference over the other 
medium, this should be a consideration in the actual 
media selection. 
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.Additional appeal for the model came because o£ 
its reaction in mechanistic and poorly designed 
instruction. It castigated equipment that frequently 
was out of service, had poor resolution, and often 
required additional simulation support by humans 
behind a curtain. Since these media limitations were 
so obvious and because human simulators were lazy, 
many ways had to be found to engineer the student's 
behavior to compensate for the limitations of the 
devices . 

Tosti and Ball felt that equipment didn't 
necessarily have to be sophisticated. That is, equip- 
ment did not need to be designed ar und the limitations 
of human behavior as the equipment engineers might see 
them. Many equipment features, such as embellishment 
of student response modes, were labeled bad because 
they attempted to replace simple behavioral tasks 
which could be more easily handled with a behavioral 
engineering approach. Thus, although an ••engineering" 
thesis, it was humanistic. 

A five-dimension classification scheme of 
presentational factors was devised as an attempt at 
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proper presentational engineering. These dimensions 
are listed, followed by the rationale: 

1. encoding forms. Data must be encoded in some 
stimulus form. Although this dimension is 
nominal, the categories can be arranged in 

a hierarchy according to an increasing 
level o£ abstraction. 

2. reponse demand characteristics of the 
presentation. This is another nominal di- 
mension which includes covert, selective, 
constructed, verbal, motor, or affective 
responses . 

3. presentational management. This dimension 
is ordinal and is ordered according to the 
relative frequency of the decision to modify 
the presentation. The activity of 
management in deciding to modify the 
presentation is made on the basis of some 
assessment of the student or his environment 
One may manage objective-criented activities 
by providing learning tasks, remedial 
exercised or enrichment activities, or one 
may manage motivation by providing high 
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probability contingency activities, 
duration. Presentation varies on this ordinal 
dimension from transient to persistent depend- 
ing upon the duration of the stimulus. Movies 
usually are conveyors of more transient 
presentation, and texts display relatively 
persistent ones. In a classroom, presentation 
by lecture is more transient than one which 
is delivered by the blackboard, 
distribution. This is the ordering, grouping, 
and sequencing of behavioral items and the 
temporal spacing between such items. 
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FIGURE 11 



Dimensions of Presentation 



The dimensions were summarized as follows: 



Encoding 



Response Demand 



Environmental Structure 



Pictorial 



Symbolic 



Verbal 



Covert 



Selective 



Constructed 



Vocal 



Motor 



Affective 



Presentation Management 



Duration 



Frequent 



r 



Frequency 
of decision 
to change 
presentation 



J, 



Fixed 
pace 



Transient 



Length of time 
the presentation 
remains intact 



1 



Persistent 



Behayiorally 
structured 



Unstructured 



Distribution 



This first section o£ tfiB chapter has attempted to 
specify^ some o£ the theoretical constructs of the model 
for media and presentation design. Hopefully, it 
conveyed some of the spirit of excitement that was 
generated by the notion that a significant contribution 
to educational methodology^ was in the making • 
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Research: Variahles 



This section discusses the transitional, developmental 
process where the theoretical framework just summarized was 
converted through a series of iterative revision into the 
final course product described in the last section of this 
chapter. It is of no small importance that some statement 
be made regaiding the process whereby a conceptual framework 
for instructional strategies and instructional systems 
design is translated into a viable operational system. 

On the other hand, it would be presumptuous for this 
section of the report to pretend to offer insight that 
usurps the careful analysis of these research efforts which 
attempted, through the most rigorious methodology possible, 
to define optimum instructional design. These efforts have 
been reported in: 

TR-6.11 - An Analysis and Evaluation of Instructional 
Methodology For A Multi-Media Course in 
Leadership, Psychology, and Management" 



Phase it Evaluation 



eport 



TR-6.12a - Report of Phase II Research Findings: 

The Design and Methodology For Research 
On The Interaction Of Meaia, Conditions 
Of 1 ns truct ion , And Student Character is tics 
For A Multi-Media Course in Leadership," 
Psychology and Management, Part I: 
Conditions of Instruction 

TR-6.12b - Report of Phase II Research Findings: 

The Design and Methodology For Research 
On The Interaction Of Media, Conditions 
Of Instruction, And Student Characteristics 
For A Mult i -Media Course In LeadershipT 
Psychology and Management, Part II: / 
btuaent unaracteristics 
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TR-6.13 - Design Specifications Document Including 

Specifications For Product And Course Design 
System Management And Evaluation Procedures 

TR-6.15 - An Analysis and Evaluation Of Instructional 
Methodology For A Multi-Media Course In 
Leadership, Psychol og^^ and Management 



Phase III Evaluation Report and, 

TR-6.16 - Report of Phase III Research Findings; 

I i^or A Multi-Media Course in Leadership, 
Psychology and Management 

However, since those documents may be not readily 
available, a summation of the findings is justified here* 
Should this discussion raise questions as to either the 
validity. of the data or the wisdom of the interpretations, 
I the reader is directed to those reports cited. 

The efforts to implement the Tosti-Ball model to 
fulfill project responsibility for determining a media 
selection procedure began as research and eval- 
uation plans [TR 6.3a] were prepared. This two part 
I document outlines the procedures and methodology to be 

employed in evaluating the proposed instructional system 
I and learning modules for the leadership management course. 

The research investigation expressed the intention of 
determining if the Tosti-Ball model of presentation 

( variables could be effectively employed as the basis for 

I 

media selection. Where possible, media would be compared, 
but since the express contention of the theoretical 
model was the importance of presentation variables, there 
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was never any concerted effort to explore the relative 
value of media ^qua media. 

The variables to 5e considered were student variables, 
task variables, and presentation design variables. 

The student variables to be considered were to be 
drawn from standard tests, such as the Edwards Personal 
Preference Schedule, the Sixteen Personality Factors 
Scale, the Ohio State Psychological, and the Strong 
Vocational Interest Blank. Other variables which were 
to be looked at for their value as performance predictors 
were the standard college entrance scores CSAT, verbal and 
math, English Composition, and Math Achievement) , and 
information derived from a questionnaire prepared for 
the students Csee Appendix F ). A detailed list of all 
items covering the student variables is available in 
Appendix A, Part One, of this report. 

While there would be an attempt to identify variables 
predicting final course achievement, the main thrust of 
the investigation was oriented toward performance with 
particular presentation variables in selected media. 
Additionally, where predicted variables could relate to 
performance with particular media, some analysis would be 
done. Overall, it was the intention to examine the 
relationship between alternative conditions of instruction 
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design for the same suhject matter to compare the rela- 
tionships to student variables and to discover inter- 
actions between method and student. A more detailed 
summary of the rationale for student variables is 
available in Part One of this final report, pages thirty 
six through forty. 

The task variables to be examined according to the 
design of the final course product were built around an 
equating of tasks to the specific learning objectives. 
An objective was defined as a description of the pattern 
of behavior or performance that a learner must demonstrate. 
The objectives or tasks were further defined to be behavioral 
in nature: i.e., the task performance would be observable 
and measurable. 

It might be noted here that dur- g the course of the 
project, some confusion persisted over the term "behavioral 
approach" and its relationship to the investigations and 
the final course structure. The behavioral approach in 
no way represents a logical model for a selection process. 
Rather does it denote a rationale for procedures and 
methodology of investigation. The behavioral approach 
suggests that decisions be made on the basis of empirical 
data derived from observable and measurable performance. 
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Thereforci in a true sense o£ the term, there is no 
relationship between the final course structure and the 
behavioral approach, although there can be a close 
scrutiny of the degree to the behavioral orien- 

tation was adhered to. For that matter, one may even 
wish to argue with the behavioral approach and declare it 
an invalid procedure for decision making in the educa- 
tional process. Indeed, there are many wrho view the 
application of behaviorism to the educational process 
with dissatisfaction. They argue that such an approach 
is mechanistic to the point of reducing human beings, 
both instructors and learners, to stimulus -response 
animals, leaving no allowances for higher values and 
operations. This ideological battle, however, is not 
an issue here. Behavioral defines an approach whereby 
a concerted effort is made to obtain performance data 
whose characteristic is measurable and observable. This 
project did not exclude other kinds of data of a more 
subjective nature and it never had any quarrel with human 
judgement. Within the rationale of the behavioral 
approach, the project staff sought, wherever possible, 
to structure the investigation and to reach conclusions 
through empirical methods* 
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This discussion of the task variables summafizes 



although it intends to provide sufficient material to 
make the efforts of the research investigators compre- 
hensible. 

The main question raised by the researchers was 
whether or not presentation variables and media have 
similar effects on the achievement of different types 
of objectives. This effort would be facilitated, 
it t'as felt, through a classification of "types of 
learning". Some effort was made to distinquish this 
term from products of learning. While there was a 
heavy reliance on the structuring and ordering of 
objectives through classifications such as Bloom's 
and Gagnc^s, it was argued that such classifications 
referred to products of learning. And, it was further 
contended, that while there is an obvious relationship 
between the content products and processes of learning, 
it was argued that the goal of the project was to 
concentrate on processes and the relationship between 
processes and products. Without these distinctions, 
it would have been necessary for a full discussion and 
analysis of task variables to include a systematic 
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I arraying o£ content-, product-, and process related 

variables-. While sucL distinctions may not have great 

I merit for generalized discussions of the educational 

process, careful investigations must necessarily limit 

■ their scope and intentions. The main question, then, to 

I summarize and restate, raised by the classifications of 

types of learning, is whether or not the presentation 

I variables and media have similar effects on the achieve- 

ment of different types of objectives. 

I The primary variables representing different classes 

I of learning, however, were the cumulative posttest 

[CPT] items developed as special norm-reference research 

I tests. These tests were developed to have an approximately 

equal number of items representing acquisition of 

I knowledge and application. Thus, for purposes of this 

investigation, two broad classifications of types of 
learning were made. Type I represented an acquisition 
of simple knowledge, while type II represented application 
of knowledge. As items designed for content validity 
with high discriminative powers, it was felt that both 
types of items, representing two distinct types of 
learning, would tap abilities in Bloom's other categories 
of comprehension, analysis, synthesis, and evaluation. 
Thus, the task variables were limited to two broad 
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^ classifications o£ types o£ learning. While it could be 

I argued that a larger repertoire o£ learning types should 

have been included in the investigations, the decision to 
restrict the types o£ learning to only two had justification, 
On consideration o£ the large scale manipulation o£ condi- 
tions of instruction, the development of tests representing 
rather large classes of types of learning was felt to be 
most in keeping with the general design of the research. 
The finding that particular presentation forms and media 
had different effects on very narrow classes of behavioral 
objectives would remove the advantage of instructional 
design at the macro level: thus, it was desirable to 
determine if substantial effects on the conditions of 
instruction could be demonstrated using tests which 
included items representing several types of learning. 
Additionally, the finding of different effects with broad 
classes of objectives would support the procedure for 
segregating instruction on particular content according 
to those broad types of learning. Thus, the preparation 
of instructional units could proceed with relatively easy 
large scale control of presentation and media as appropriate 
to the two major classes of learning. There would be no 
compelling requirement for major analysis at the micro 
level. Such a procedure seemed the most practical, resol- 
ution of the condition set forth in the original contract 
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specifications; namely, that material development proceed 
from an experimental mapping of different media to 
different objectives vitRin each, unit of instruction. 
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Research Plans and Implementation 

At a more detailed level o£ planning for the imple- 
mentation o£^ the research; effort the following questions 
were framed* 

a# Are substantial effects on student achievement 
produced by manipulation of presentational 
variables at the macrotaxonomic level as 
conceived by Tosti and Ball (1969)? 

b. Are substantial differences in student 
achievement produced between different media 
delivering the same presentation , when 
measured over segments of material typical of 
a unit of instruction in most educational 
systens? 

c. Are variations of conditions of instruction in 
the presentation design domain of greater or 

NLesser importance than variation in the media 
domain? 

d» Are the effects of presentation and media 
variables generalizable over different types 
of instructional objectives, or are different 
effects produced in relation to the acquisition 
and application of knowledge? 
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e. Are effects of particular presentation conditions 
and media similar for students varying according 
to established standardized tests of individual 
differences^ or do the optimal conditions of 
instruction differ for different students? 
There would be problems attending to all these 
objectives within a single ongoing course. It would be 
necessary to arrange the experimental manipulation of 
materials and measurements so as to avoid the mutual 
entanglements of the effects of different experiments. 
Substantial variation of the level of difficulty in 
specific course content and test items required control 
to prevent obscuring of experimental effects. The small 
number of students available for enrollment in a develop-* 
mental course required that specific techniques for 
reducing random variation be employed to increase the 
precision of the experimental comparisons^ but which would 
not interfere with the investigation of individual 
differences in relation to experimental variables. On the 
other hand 9 the student could not be burdened with complex 
experimental procedures which might handicap his overall 
achievement. The ability of the investigators to design 
a plan which would avoid the pitfalls just me^itioned^ 
yet achieve answers to th.ose questions just statedi is a 
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tribute to both their professionalism and their ingenuity. 
Full details on the experimental design and specific 
research plans are availalile in TR 6«12a, Phase II ,. 
Research Findings » ?att 1 : The Conditions of Instruction , 
and TR 6.16, Phase III , Research Findings . For our 
purposes here in tracing the evolution from the theoretical 
construct to the final course product, the following items 
are discussed in summary fashion. 
Selection of Media 

The selection of media and presentation variables 
for the first implementation of the course was determined, 
in large measure, by the requirements of the experimental 
design, capacities needed for delivery of instructional 
presentation, and the diversity and flexibility expected 
of an individualized multi-media course. An extensive 
discussion on these conditions is available in the 
documents just cited. These reports explain the problems 
inherent to experimental design which influence both the 
selection, placement, and sequencing of media. In retro- 
spect, some have argued that the selection of media was 
arbitrary, or at least so biased by experimental require- 
ments that it had little value in either helping to design 
a final course system, or to provide a universal media 
selection strategy for a multi-media course model. These 
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arguments seem predicated on tlie notion that the 
research efforts, regardless of the limitation of the 
variables examined, should yield a universal algorithm for 
media selection and course design. Secondly, there seems 
to be an effort to place developmental research in a camp 
quite apart from standard course development. One comment 
that was heard raised the cora5)4aint, "Yes, the 
research is fine, but vhen will we start on the course?" 
Such a comment failed to understand that the initial course 
design was set up to test the largest number of variables 
and conditions of instruction possible with the express 
intent of continually refining the instructional variables 
through successive approximations to yield the optimum 
course structure. Under no conditions was it the intention, 
nor could it be justified, that the research and evaluation 
efforts would establish a set number of principles upon 
which course fabrication would begin. Instead, almost 
five hundred modules of instruction were prepared with 
the express purpose of systematically eliminating those 
conditions of instructions which failed to yield the highest 
possible performance and which would allow restructure or 
incorporation of those conditions of instruction which 
proved valuable. The discussion to follow traces the 
successive iterations of the experimental test runs to 
arrive at the final course system. 
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Initial Results and Revision of Model 

The first run of the course in the spring of 1970 
attempted to gather data for subsequent analysis on the 
various conditions of instruction, their associated media, 
and student characteristics which might correlate to 
performance. A summary of the research plan for the spring 
1970 run is shown in Figure 12. There will be no discussion 
here of details on the chart other than to note that a wide 

variety of media and presentation variables were under 

( 

scrutiny* Again, details are aval"* able in documents 
previously cited* The outcome of these investigations 
yielded certain conclusions which were the ba::is for 
change in the second implementation in the fall of 1970* 
The first conclusion derived, which was to be later 
corroborated in the second run or iteration, was that 
there was no strong evidence to support the Tosti-Ball 
theory on the importance of presentation variables. While 
some differences were found, the overall trend from this 
series of studies was no significant differences were 
found when the medium was constant and the presentation 
design was varied. There were some exceptions. In the 
linear text the high response demand frequency condition 
was consistently superior to the low demand frequency 
condition. Further, there was a trend which indicated 
that overt selected response condition was slightly better 
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FIGURE 12 cont. 
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than either overt spoken or covert. In regard to management 

frequency, there was little conclusive evidence 

related to the manipulation of this presentation variable. 

Remediation variables likewise seemed to have no effect 

on the student's performance. Peer interaction as still 

another variable, yielded little significance. There 

was one surprise, almost in contradiction to the Tosti-Ball 

concept. Differences were found in favor of audiotape over 

videotape lecture format. 

While there was some pressure to replicate the exact 
experimental design in the second run [Fal] 1970] to cross- 
validate these findings, there was sufficient evidence at 
this point in time to make some alterations to the design 
which might move it forward to the final selection process. 
Figure 13, indicates the changes which were made in the 
experimental design which attempted to refine these 
selections. First, videotapes were dropped, partly because 
of the experimental findings, partly because revision costs 
approached the prohibitive level and other performance 
indicators suggested a cost effectiveness disadvantage. 



logistically feasible for large numbers of students. Peer 
interaction, as a variable, was no longer under consider- 
ation, and it was :*eplaced by an advanced organizer technique, 
the content map, as a variable in the instructional package 



and partially because videotape equipment would not be 
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Presentation Design FIGURE 15 

Second Run 
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FIGURE 13 cont. 
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which, migh-t facilitate a learning process. To further 
examine the desirability of media dependent presentations, 
printed versions of the CAI sections and selected auaio- 
tape scripts were prepared for comparison against the 
standard presentations. 
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' Results of Second Iteration and Final Selection 

Once again the overriding conclusion from the fall 
run came from data which did not support the Tosti-Ball 
theory on the importance of presentation design charac- 
teristics. Hovrever, the researchers were very careful 
to note that this was the first experimental effort 
examining their basic thesis and carefully avoided 
generalizations which may not hold true for other 
populations and experimental conditions. The lack of 
significant differences, it should be noted, should in 
no way be equated to failure. It is a benefit to 
education to find that sophisticated hardware presentations 
are no more effective than less expensive paper formats 
properly constructed. And, if this series of studies 
tends to retire from further consideration the importance 
of presentation variables, it tends to free the way for 
further study of previously little noted variables in 
instructional development. The course was phenomenally 
successful and some would suggest that the failure to 
find major differences in the presentation variable was 
the direct result of the fact that almost all the materials 
were successful in meeting criterion performance. 

These points, however, did not diminish the ongoing 
responsibility of refining the media and 4>resentation 
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selection process to derive the final course system. 
At the time that the fall research, run data was 
beginning to indicate a confirmation of many previously 
established differences or no differences, work began 
on preparing the final course product and system. 
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Perhaps the major decision relative to media 
selection for the final course system was the speci- 
fication of a printed format as an alternate to the 
audiotape presentations. The form selected was the 
syndactic text which had proved both successful and 
popular and which lent itself more readily to the 
concept of individualization. The second decision 
pertained to the specification of those final modules 
which would represent the main line of instruction in 
the final course system. Despite the fact that the 
overall concept of the Tosti-Ball thesis on presentation 
variables did not prove valid for this project, a 
combination of presentation variables exhibited a trend 
which justified their inclusion in the final module 
selection for the course system. High response demand 
frequency [i.e. a question every frame] with overt 
selected response [i.e. the student physically identifies 
the right answer by writing it or circling from appro- 
priate choices] with medium management frequency were the 
three variables that were incorporated into the final 
modules for each segment. Figure 14 indicates the 
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modules which were selected for the final course 
sy^stems design* Where these modules did not conform 
to high response demand frequency^, overt selected 
response, and medium management frequency, adjustments 
were made accordingly. This table is identical to 
Figure 13 which depicted the media and presentation 
variable options for the fall *70 run of the course* 
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Final Course Content 



*^^® course is composed of 12 parts comprising 59 
segments. The titles of the parts and segments are listed in 
7- . Table 7. 

^^^^ 1 ^he course, Overview of Leadership, briefly 
F presents theoretical bases of leadership, and evolves there- 

^ from a view of leadership as a dynamic interaction process 

[ involving a leader, his followers and an environmental 

J situation. Part 1 further delineates the use of the basic 

principles of psychology and management as components of 
I naval leadership. Acceptance of a traditional deep sense of 

moral responsibility as an essential trait of a naval leader 
[ is stressed here and intermittently throughout the course. 

The naval leader will most often be dealing with groups 
of individuals, his subordinates or followers, so he should be 
I aware of the effect of an individual's membership in a group. 

A fundamental knowledge of individual psychology and group 
I dynamics is provided in Part 2, Individual Behavior, and 

Part 3, Group Dynamics. 

No leader, regardless of his intelligence, or the 
j originality or brilliance of his ideas, can be successful 

until he can communicate those ideas and directions effectively. 
! Part 4, Achieving Effective Communication, is designed to 

impart to the midshipman a knowledge of the principles of 
communication so that he win be effective in his role as a leader. 
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TABLE 7 



COURSE CONTENT 



PART ONE; Overview of Leadership 
Segment 1: Concept o± Leadership 

Segment 11: Standards of Leadership in the Naval Service 



PART TWO; Individual Behavior 
Segment 1: Introduction to Psychology 

Behavior and Its Observation 
Learning 

Factors Affecting Learning 
Attention and Perception 
Motivation 
Conflict 

Abnormal Behavior 
Personality 



Segment 11: 
Segment 111: 
Segment IV: 
Segment V: 
Segment VI: 
Segment Vll: 
Segment Vlll: 
Segment IX: 



PART THREE: Group Dynamics 
Segment 1: Characteristics of Groups 

The Relation of the Leader to the Group 
Group Interactions 

Conformity as a Factor of Group Behavior 
Relation of the Individual to the^Group 



Segment 11: 
Segment 111: 
Segment IV: 
Segment V: 



PART FOUR: Achieving Effective Communication 



i>egment 1: 
Segment 11: 
Segment 111: 
Segment IV: 
Segment V: 
Segment VI: 
Segment Vll: 



Importance of Interpersonal Communication 
Types of Communication 

The Communication Process (Receiver and Barriers) 
The Communication Process (Sender and Feedback) 
Formal Communication and Its Dimensions 
Informal Communication 
Communication Under Battle Situations 



PART FIVE: 
Segment 1: 



Segment 11: 
Segment 111: 
Segment IV: 
Segment V: 
Segment VI: 
Segment Vll: 
Segment Vlll: 
Segment IX: 
Segment X: 



Military Management 

Introduction to Management and the Management 
Process 

Decision Making and Creativity 
Objectives 



Planning 
Organizing: 
Organizing: 
Organizing; 
Directing 
Controlling 
Coordinating 



Principles 

Structure 

Charting 



and Process 
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TABLE 7 Continued 



PA . \ : ly Authority and Responsibility 

SF4nr./'^ It Concept ot' Authority 

Segment 11: Why People Accept/Resist Authority 

Segment 111: Delegation of Authority; Line-Staff Relationship 

Segment IV: Responsibility 



PART SEVEN: 



begment 1: 
Segment 11: 
Segment |11: 
Segment IV: 

Segment V: 



PART EIGHT: 
Segment 1: 



Leadership Behavior and Style 
Leadership Behavior 



Leadership Style 
Determiners of Leadership Style 
Determiners of Leadership Style 

and The Situation 
Participative Leadership 



The Leader 
The Group 



Senior-Subordinate Relationships 

Organizational Structure § Social Distance in 



Segment 11: 
Segment 111: 

Segment IV: 
Segment V: 
Segment VI: 



Senior-Subordinate Relationships 
Officer-Enlisted Relationships 
Assumption of Command and Formal § Informal 

Leader Relationships 
Introduction to Counseling 
The Counseling Process 

Relations with Seniors and Contemporaries 



PART NINE: Morale - Esprit de Corps 
Segment 1: Morale 

Segment 11: Group Solidarity and Esprit 



PART TEN: Discipline 

Segment 1: Introduction to Discipline 



Segment 11: 



Development and Maintenance of Discipline 



PART ELEVEN: Personnel Evaluation 

Segment 1: The Role of Evaluation 

Segment 11: Enlisted Performance Evaluation 

Segment 111: Officer Evaluation 



PART TWELVE; Applied Leadership 



Segment 1: 
Segment 11: 

Segment 111: 
Segment IV: 



Measurement of Effective Leadership 
Generally Recognized Characteristics of an Ef- 
fective Leader 
Techniques of Assuming Command 
"That's an Order 1" 
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A naval leader will, in his daily routine, be working 
with men, money, material and time. The effective use of 
resources available to him can best be achieved by the leader 
who can apply the Management Process. Part S, Military 
Management, provides the midshipman with the fundamentals of 
the Management Process. 

The latter parts of the course concentrate more specifical 
on those aspects of leadership which are peculiar to the 
military. 

The naval leader, as an appointed leader, has an official 
authority foundation in his commission or appointment. Moral 
responsibility for mission accomplishment, subordinates' 
welfare and general efficiency is perhaps more intimately 
related to the military leader than to leaders in other areas 
of endeavor. The general theories of authority, the 
responsibility coincident with authority, and proper delegation 
of authority are treated in Part 6, Authority and Responsibility 

A leader's behavior and style are variables in any 
leadership situation. The spectrum of leader behavior, the 
variety of styles, and the causes/effects of the variations 
are explained in Part 7, Leadership Behavior. and Style. From 
such knowledge, the midshipman can formulate a personal 
concept of possible behavior and style for himself. 

The effectiveness of a leader and the influence he can 
exert on his followers depend to a large extent on the 



relationship he establishes with the subordinates. Building 
on the foundation acquired in Parts 2 and 3, Part 8, Senior- 
Subordinate Relations, delineates specific guidelines for 
the leader in his relationship with his subordinates. The 
counselor role of a leader, among his many responsibilities 
for the welfare of his men, is emphasized. 

Part 9, Morale, treats the subject of morale and esprit 
de corps, their importance to an effective organization, and 
how a leader's actions influence the individual followers 
and the group collectively. 

The ability to instill a willingness in one's self or 
one's followers to act correctly under adverse conditions is 
essential for a naval leader. The most desirable situation 
exists in a military organization when each member willingly 
chooses to do the right thing because it is the right thing, 
rather than doing it only because of fear of the consequences 
of not doing it. Such is the nature of discipline as treated 
in Part 10, Discipline the positive aspect of discipline 
as opposed to the negative or punitive connotation. 

An organization, properly managed by appointed leaders, 
prepares for the continuity of leadership of . that organization. 
Essential to the selection and training of leaders, as well 
as the maintenance of high standards is a personnel evaluation 
process. Part 11, Personnel Evaluation, provides the midship- 
man with a knowledge of the system by which he is himself 
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evaluated and introduces him to the role of evaluator. which 
he must fill as a leader. 

Part 12, Applied Leadership, serves as a review and 
synthesis of the course. Situational examples are provided 
for the midshipman to observe the application of material. 

The twelve parts identified in Table 12 represent major 
content areas. Each part is divided into varying numbers of 
segments, ranging from as few as two in Part 1 to as many 
as ten segments in Part 5. A segment is a collection of 
learning objectives closely related by content which can be 
studied in about an hou r's time. 

final Course S tr ucture and Operation 

The Course Activities .Chart on nage IPfi ??hows the sequence 
in which the 59 segments and all other events will occur. The 
chart should be read starting at the upper left (Introduction 
and Pretest) and following the numbering of the segments from 
1.1, 1.2, 2.1 ... 12.4. Each segment bears a dual number - 
the first is the part number and the second is the number of 
the segment within the part. Each square in the chart 
represents a segment with the segment number given .-ibove the 
square. The identification of the type of material as well 
as a time estimate for that segment is inside the square. The 
time estimate for each segment is the average number of minutes 
needed by the student to work through the instructional materials, 
including the Progress Check and remediation (if necessary). 

The same instructional content may be available in more 
than one media. In this case, two boxes are shown under a 
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segment number in the chart and the media options are identified. 
For ease of distribution and handling, several segments of like 
media have been bound into a single vola.e. The choice of media 
^ is up to the student, however, for logistic and administrative 
reasons, the student selecting a media option for a given 
segment should continue with all other segments in that same 
volume, rather than selecting a different media for an adjacent 
segment. For example, if a student uses the Audiotape/Panel 
Book (AT/PB) materials in 2.2, he should continue with the 
Audiotape/Panel Book version through Segment 2.5. Table 8 
shows the volume numbers, segments included in that volume, 
and media options for identical instructional content, where 
available. 

Other events detailed on the Course Activities Chart 
include depth core (DC) , administrative tests and the minimum 
pace to be maintained by the student in the course. The 
depth core is an in-class discussion session with required 
attendance which aims at looking "in depth" at content .;overed 
in previous segments. Although eight depth core meetings 
have been scheduled in the Course Activities Chart, the instructor 
may supplement or delete the scheduled depth. core sessions. 
The student is responsible for checking the board for depth 
core schedules and for attending the meetings. Generally, 
the depth cores have been developed on the assumption thai 
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TABLE 8 

SEGMENTS BY VOLUME AND MEDIA EMPLOYED 



Volume 1 Segments 
Number | Included 
i ■ 


Prime 
Media 


Alternate 
Media 


Volume 
Number 


I i 1.1 

! 1.2 


Syndactic text 
discussion booklet 






ii-A „' 271 

; 2,2 
j 2.3 
i 2.4 
i 2.5 


iiyndactic text 
Audio tape/Panel book 
Auaiotape/Panel book 
Audiotape/Panel book 
Audio tape/Panel book 


Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 


II-B 


XX -u 


; ^. 0 
2.7 
2.8 
2.9 


oynaacnc text 
Syndactic text 
Syndactic text 
Syndactic text 






Ill-A 


■ 3.1 
3.2 
3.3 
3.4 


Auaiotape/Fanei book 
Audiotape/Panel book j 
Audiotape/Panel book 
Audiotape/Panel book 
Syndactic text 


byndactic text 
Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 


lil-B 

1 


IV-A 


4.1 
4.2 
4.3 
4.4 


Linear text 
Linear text 
Linear text 






Aud iotape/ Intr ins i - 
cally Programed 
booklet 


Tape Script § 
Intrinsically 
Proizramed book 


IV- 

Script 

let 


IV-B 


4.5 
4.6 
4.7 


Audiotape/ Intrinsi- 
cally Programed 
booklet 

Audiotape/ Intrinsi- 
cally Programed 
booklet 

Audiotape/ Intrinsi- 
cally Programed 
booklet 


Vape Script ^ j 
Intrinsically 
Programed booklet 

Tape Script § 1 
Intr insicnlly 1 
Programe-' booklet 

Tape Script § j 
Intrinsically j 
Programed booklet 


V-A 


5.1 
5.2 
5.3 
5.4 
5.5 
5.6. 


Syndactic text 
Syndactic text 
Syndactic text 
Linear text 
Linear text 
Linear text 






V-B 


5.7 
5.8 
5.9 
5.10 


AUdiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text \ 
Syndactic tr. t 
Syndactic text 
Syndactic text 
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TABLE 8 (Continued) 



Volume 
Number 


Segments 
Included 


Prime 
Media 


j Alternate 
Media 


Number 


.! VI -A 


6.1 
6.2 


Linear text 






• VI -B 


6.3 
6.4 


Linear text 

O/iiUciL. tic cex c 




VI I -A 


7.1 
7.2 
7.3 
7.4 
7.5 


Audiotape/Panel book 
Auaiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text 
Syndactic text 


VII-B 


syndactic text 
Syndactic text 
Syndactic text 


vn-c ■ 


VIII-A} 8.1 
j 8.2 
1 8.3 


Linear text 
Linear text 
ijiiiea.x cexc 






VIII-B' 8.4 
1 8.5 
j 8.6 


Syndactic text 
Syndactic text 
Svndactir tAvt* 


IX-A j 9.1 
j 9.2 


Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text 
Syndactic text 


IX-B 


X 


10.1 

j 

10.2 


Audiotape/ Intrinsic 
cally Programed 
booklet 

Audi otanp/Tn^Y^-iT^e-i - 

cally Programed 
booklet 


Tape Script § 

Programed boo! 
Tape script § 
Intrinsically 
Programed book 


X Script 
:let 

let 


. XI 


11.1 
11.2 
11.3 


Syndactic text 
Syndactic text 
Syndactic text 








12.1 
12.2 
12.3 
12.4 


Computer-Assisted 

Instruction 
Computer-Assisted 

Instruction 


Intrinsicall- 
Programed ' ;k 

Intrinsicaii >• J 
Programed book 


XI I -A 
let 
Xll-B 
let 


Computer-Ass isted 
Instruction 


Intrinsically |XII-C 
Propramed hooVi*»t 


Computer-Assisted 
Instruction 


Intrinsically jxiI-D 
Programed booklet 
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the students will have completed all segments which, 
precede the depth core in the Course Activitites Chart. 

To gain maximum benefit from the group discussions, therefore, 

the student should ensure that Ke has finished the requisite 

materials which precede a depth core meeting. The instructor 

has the option of grading his students on their participation 

in depth core sessions. 

Administrative tests are USNA tests on which the student's 
grade will be based. Seven administrative tests at two-week 
intervals are scheduled in the Course Activities Chart. The 
segments covered in each test are listed in the chart next to 
each test. The instructor has the option to override the 
testing schedule in the chart by testing less frequently (such 
as every four weeks) or testing over different segments of 
materials. As with depth core sessions, advance notification 
of tests will be given by the instructor or posted on the board 
outside The Course Administrator's Office. A final examination 
on the entire content of the course will be administer ^^d during 
the regular exam week. 

In this individualized system a great deal of respon- 
sibility for the timely completion of materials will be placed 
on the student* To aid the student in properly allocating his 
study time, a minimum pace has been indicated in the Course 
Activities Chart by grouping the segments into one week's work 
of five-six hours. This is the least amount of effort expected 
of the student if he is to be prepared for depth core meetings 
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and administrative tests. There is no upper limit to the speed 
with which the student may proceed through the course, indeed 
the student may find it helpful to complete the course materials 
long before the semester is over. 

Although not indicated in the Course Activities Chart, 
another form of instructional material, the Enrichment Unit, 
will be available to the students. Enrichment Units are related 
to, but not essential to, the mastery of the overall objectives. 
The student whose interest is stimulated should discuss the 
options available with his instructor. 



Instructional Media Employed 

This section describes the types o£ materials the student 
will encounter and gives directions on how the materials are to 
be studied. 

The Linear Text should present no difficulties for the 
first-time user of programed instruction. A linear text is 
constructed in a series of frames; a frame may consist of a 
small bit of teaching information plus a question demanding a 
student response, or solely of a question pertaining to prior 
teaching. The student studies the linear text by beginning 
with frame 1 and working sequentially through the frames, 
verifying his frame responses by unfolding the program Frame 
Answer Sheet (Sample in Appendix G) at the back of the 
instructional volume. 

i The Syndactic Text has two components: a Summary and a 
Programed Sequence. The student begins by reading a brief 
statement of a given body of learning material (Summary 1). 
He then takes Summary Pre-Quiz 1 (generally five questions) to 
test comprehension and retention of the material just read. 
The student checks his responses against the correct answers 
given in the text. 

If he has answered all questions correctly, he proceeds 
to the next summary, Summary 2, and to Summary Pre-Quiz 2, and 
so on. If the student has incorrectly answered, one or more 
questions on Summary Pre-Quiz 1, he is directed through 
Programed Sequence 1. This sequence of linear frames provides 

202 



detailed learning of the material presented in the Summary. As 
he works through the Programed Sequence, the student may check 
his responses to questions in the frames by unfolding the 
Program Frame Answer Sheet (Appendix GO which is bound at the 
back of the volume. When the student has finished Programed 
Sequence 1, he will take Summary Post-Quiz 1, and check his 
answers against the correct ajiswers given in the text. He 
then goes on to Summary 2 where the same procedural cycle is 
repeated. 

The following flow chart will assist those who may have 
difficulty with this instructional method: 




The Intrinsically Programed Booklet is commonly referred 
to as a "scrambled" book. By design, the student cannot work 
sequentially through the text. Rather, a page will present 
instructional material and direct the student to another page 
where a multiple-choice question is posed. 
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There are two types o£ questions in the Booklet, For the 
first type of question, the student selects his response, 
checks it against the correct answer provided on the Program 
Frame Answer Sheet, (Appendix G.)f and follows the directions 
printed on the bottom of the question page to proceed to the 
next set of instruction material. 

The other type of question is called a "branched" question 
because it branches the student to different pages depending on 
the response he selects. There is no blanket direction printed 
on the bottom of this question page - instead each possible 
response sends the student to a different page. The student is 
to select his response, and turn to the page indicated on the 
question page for his particular response. On this new page, 
his response will be analyzed. If the response is totally 
incorrect, the student is directed back to the question page to 
select another response. If his original response is wholly 
or partially correct, the student is directed to a new set of 
instructional material. 

The Intrinsically Programed Booklet is to be used in con- 
junction with either an audiotape or the written version ( script ) 
of an audiotape. If a tape is used, the student will listen to 
the first portion of the tape and follow directions on the tape 
to turn to a page in the Intrinsically Programed Booklet. He 
will work through part of the Booklet until he is instructed to 
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read the next portion of the tape* If the student is using the 
script instc*ad of the tape, he will read the first portion of 
the script and follow directions in the script to turn to a page 
in the Intrinsically Programed Booklet. He will work through 
part of the Booklet until he is instructed to read the next portion 
of the script. 

The explanation of the procedures to follow in working 
through an Audiotape/Intrinsically Programed Booklet or an 
Audiotape Script/Intrinsically Programed Booklet may appear 
complex, but the actual use of these materials will be easy be- 
cause the student receives clear directions on every page as to 
the route he is to take through the materials. 

If the student fails to attain 80% correct on the first 
trial of the Progress Check, he must go through a remediation 
cycle. The remediation material is the script which accompanies 
the Intrinsically Programed Booklet. 

The Audio tape/Panelbook media inix is another type of 
learning material the student will encounter in this course. 
The instruction is delivered via an audiotaped lecture which is 
supplemented by a panelbook containing illustrations, charts and 
questions. The student listens to the tape until the lecturer 
instructs him to "turn to Item 1" or "answer Question 2" in the 
panelbook. The student may turn off the tape recorder if he 
needs more time to answer the question. Answers to the panelbook 
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questions are found on the Program Frame Answer Sheet (Sample: 
Appendix G.) • 

If the student does not score 80% correct on the first 
trial of the Progress Check, he must remediate. In the case of 
the audiotape/panelbook segments, the remediation material is 
the written version of the tape, the audio script. The script 
is bound in the volume following the panelbook and Progress Check 
for each segment. 

Computer Assisted Instruction is a media option in the 
last four segments of the course. All instruction is delivered 
via computer terminals and the student enters into the terminal 
his responses to questions ill the material. The Progress Checks 
are in the usual printed form but remediation is done ••on line" 
via the computer. The student will receive detailed instructions 
from the computer center on interacting with the terminal. 

An alternative to the medium of Computer Assisted Instruc- 
tion is a paper version incorporating the Computer Assiste d 
Instruction Script and Intrinsically Programed Booklet , For an 
explanation on its use, the student should read the discussion on 
the use of the Audiotape Script/Intrinsically Programed Booklet. 

Regardless of the type of instructional material, each 
segment will have the same general format. There is a foreword 
for each segment, followed by the instructional materials. At 
the end of the instructional materials there is a Progress 
Check (PC) consisting of approximately 10 items. After the 
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Progress Check is a page entitled Progress Check Answer 
and Remediation Form (Sample; Appendix H) . Materials 
to be used as remediation (if different from the original 
instruction) follow the Progress Check Answer and 
Remediation Form. This sequence is repeated for each 
segment within a volume. 
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DeptJi Core 

Tlie overall design of the course vras intended to 
provide a full, yet flexible series of educational 
experiences that would insure optimum possible benefit 
for each student in the course. Hence, depth core units, 
enrichment modules and remedial lessons were developed to 
round out the basic core instructional segments^ 

The fifty nine core segments basically attend to 
the cognitive domain of the learner. It was generally 
believed that a course in leadership, psychology, and 
management should attempt to address the affective domain, 
especially since the interpersonal dynamics in leadership 
situations is so vital. Over the course of the contract, 
a series of seventeen depth core units were developed 
which served two purposes in the course. The first is 
the integration of relatively discrete skills into higher 
order learning experiences. Secondly, the depth core 
units, constructed primarily for a group mode, allow 
for directed interaction between the group and the 
instructor - not easily possible in a self-paced 
individualized course. 
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The depth core units, while compiled in a single 
syllabus, are independent of each other. They all have 
the following section format: 

1. Purpose of presentation 

2. Statement of intention 

3. Objectives o£ presentation 

4. Approach to presentation 

5. Student handout 

6. Discussion questions and responses 

7. Evaluation and procedures. 

■^^^ statement of intention , objectives of presenta - 
tion and the approach to presentation sections give the 
instructor the rationale, goals, and procedures for 
implementing the depth core unit. The student handout 
consists of either a script, narrative, or problem state- 
ment which the student works through either by himself 
and/or with other class members. The discussion questions 
and responses section gives the instructor some specific 
methodology for inductively bringing the group to either 
new insights or behavior. While there are no right or 
wrong answers, possible answers, which have a validity in 
the theoretical concept of the depth core, are offered. 
The evaluajdon procedures section suggests some ways by 
which the instructor can rate the performance of the group 
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or the iniividual. 

In general, there is an attempt to give practical 
experience in applying the cognitive principles o£ the 
course, to allow an atmosphere for insight into the dyna- 
mics of leadership, and finally, to allow some self- 
assessment on the individual student's predilection to 
have his effectiveness influenced by the variabilities 
o£ the group, the situation, and his own personality. 

Table 9 lists the Depth Core Units available to 
the instructor. 

Appendix I is a sample of a Depth Core Unit. 
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Depth Core # 1: 

Depth Core # 2: 

Depth Core # 3: 

Depth Core # 4: 

Depth Core # S: 

Depth Core # 6: 

Depth Core # 7: 
Depth Core # 8: 
Depth Core # 9: 

Depth Core #10: 

Depth Core #11: 

Depth Core #12: 
Depth Core #13: 



Table 9 
Depth Core Units 
Perception "g[nd Leadership 
Brainstorming and Group Pressure 
Leavitt-Mueller Experiment 
Achieving Effec^-ive Communication 
Preparing a Plan 

Allocation of Resources in Crisis 
Situation/NASA Film 

Group Ordering of Critical Factors 

Counseling Role Playing 

Responsibility of Junior Officer 
Counseling 

Adjustment and Maladjustment/ 
Mental Health and Therapy 

Design of New Enlisted Performance 
Evaluation Form 

Dropped from the syllabus 







Race Relations 




Depth Core 


#14: 


The 


Drug Problem and 


Drug Awareness 


Depth Core 


#15: 


The 


Leader and the Situation 


Depth Core 


#16: 


The 


World of CHARLIE 


Company, Part I 


Depth Core 


#17: 


The 


World of CHARLIE 


Company, Part II 
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Enrichment Modules 



Westinghouse Learning Corporation, in its efforts 
to bring innovative learning materials to the project, 
commissioned Structural Communications, Ltd. of London, 
England to develop eight units of materials based on a 
new and proprietary technique of discussion materials. 
The Structural Communications materials were designed 
and developed to coincide with eight of the basic 
instructional units in the core course and were so con- 
structed as to allow individual or group activity. The 
fundamental philosophy of Structural Communications is 
that the students, either individually or collectively, 
can be brought to greater understanding regarding the 
subtleties of concepts through continual dialogue with 
the authors. The philosophical foundation for this 
approach is best described in (The Progress of Educational 
Technology, Bennet and Hodgson, 1968). .In laymen's terms, 
Structural Communications postulates that students can 
be brought to ascending levels in the understanding of 
complex structures through non- linear, i.e. program or 
sequential, strategies. 

The basic strategy of a Structural Communications 
unit is to have the student work through the orientation 
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of the central focus or unifying theme which dominates 
the problems presented in a brief series of modules. 
Each successive module contains a problem of consideration 
which expl»^res one particular facet of the central thesis. 
It is stated briefly and pointedly. The student then 
refers to a matrix of statements, usually twenty four 
in number, which the student must then examine to deter- 
mine the appropriate relevance or irrelevance to the 
problem he is considering. After making his decision, 
he may immediately turn to the discussion section which 
analyzes the skill and eliminate or choose appropriate 
statements from the matrix. If he is working with a group 
he may discuss his individual choice with others to 
arrive at a group consenst^s before proceeding to the 
discussion section. As the student moves from module to 
module, his total grasp of the elements of the theme 
and their interaction becomes greater. The end product 
is a decidely enlightened student who has been subtly 
tracked through several levels of abstract reasoning. 

When Structural Communications was first intro- 
duced on a trial basis, the optimum condition demanded 
that students work through all modules of the unit in 
a single session. Since this meant nearly three hours of 
discussion and dialogue, student reaction was not alv^ays 
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favorable. While the dialogues and ideas were interesting, 
the students were constrained by time. In the revised and 
final product, both the individual and the group mode 
is permitted. Additionally, modules within a unit can 
be worked through separately - hence more convenient time 
blocks can be accommodated. 

Table 10 lists. the structural communications units 
and sub-modules. 

Space does not permit an enclosure of a structural 
communications module, even as an appendix to this report. 
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Table 10 



Enrichment Units 
Structural Communications 



Enrichment I: 

Module i: 

Module 2 : 

Module 3: 

Enrichment II: 
Module 4 : 
Module 5 : 
Module 6 : 
Module 7 : 

Enrichment III: 
Module 8: 
Module 9 : 
Module 10: 
Module 11: 

Enrichment VI: 
Module 12 
Module 13 
Module 14 



Enrichment VIII: 
Module 15: 
Module 16: 
Module 17: 
Module 18: 

Enrichment XA: 
Module 19: 
Module 20: 

Module 21: 

Enrichment XB: 
Module 22: 
Module 23: 



Module 24: 
Module 25: 



Concept o£ Leadership 
Leadership in a Civilian Organization 
Riot Leadership 
Naval Leadership 

Problems in Individual Behavior 
Case of the Missing Ensign 
Case o£ the Angry Seaman 
Case o£ the Liquidated Project 
Case o£ the Antarctic Accident 

Group Dynamics and the Chain o£ Command 
Group Dynamics in a Naval Air Squadron 
Recognition o£ Structure in the Group 
Social Climates £or Leadership 
Engineering a Group Situation 

Problems o£ Leadership 
The Case o£ the Missing Ham 
The Case o£ the Unmoved Files 
The Case o£ the E££icient Ensign 

The Motivation o£ Subordinates 
Motivation 
Taking Command 
Logistics 
Leadership 

Discipline and the Group 
The Prison - the unwilling inmate 
The Monastic Community - the voluntary 

prisoner 
The Scattered Organization 

Discipline and the Individual 
American P.O.W.'s in Korea 
American P.O.W.'s in Korea 

(Captor* s Approach to Eroding P.O.W. 
Discipline) 
Tension on Board a Destroyer 
Student Unrest 
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TABLE 10 cont. 



Enrichment XII 
Module 26 
Module 27 
Module 28 



Learning to Lead 
The Upward Mobile Ensign 
The Ideal Ensign 

Three Perspectives o£ the Junior Of£i 
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Remedial Procedures 

^ In the final course product, provisions for reme- 
diation are diverse, yet practical. Table 11 indicates 
the variety and type of remediation sources. 

In general it should be noted that remedial materials 
for hardware oriented presentations is available in a 
persistent medium. Thus, where audio tapes or CAI might 
fail to bring students to criterion performance, a 
programmed version of the material or the script of the 
tape itself is available. Where hard cover materials 
constitute the initial presentation, the students are 
rerouted through the same material. When the primary 
source is a syndactic text, the student has an option of 
either reviewing the summary statements or working through 
the programmed sequences. 

The actual mechanism for determining the necessity 
of remediation is the Progress Check Answer and Remediation 
Form located at the end of each segment Csee Figure 15). 
The student records his answers on a separate sheet and 
checks them against the answers. If he misses more than 
two of the items, he is required to review the entire 
segment. The location for specific trouble areas is listed 
in the remediation reference column. 
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TABLE 11 

REMEDIAL MEDIA 



1 

Volume Segments 
Number | Included 


Initial Instructional 
Media 


Remedial 
Media 


Vo lume 
Number 


■ 

I I 1.1 
i 1.2 


Syndactic text 
Discussion booklet 






II-A ' 2.1 
2,2 

i 2.3 
2.4 

' 2.5 


Syndactic text 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 


II-B 


II-C 


: 2.6 
2.7 
2.8 
2.9 


Syndactxc text 
Syndactic text 
Syndactic text 
Syndactic text 






III-A 


3.1 
3.2 
3.3 
3.4 
3.5 


Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Syndactic text 


Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 
Syndactic text 


III-^ : 


! IV-A 

t 


4.1 
4.2 
4.3 
4.4 


Linear text 
Linear text 
Linear text 






Audiotape/ Intrinsi- 
cally Programed 
booklet 


Tape Script § 
Intrinsically 
Programed hooV 


IV- 

Script 
:let 


IV-B 


4.5 
4.6 
4.7 


Audiotape/ Intrinsi- 
cally Programed 
booklet 

Audiotape/ Intrinsi- 
cally Prooramed 

ibooklct 

Audiotape/ Intrinsi- 
cally Programed 
booklet 


Tape Script § 
Intrinsically 
Programed boot 

Tape Script f, 

111 cx xiid V L 1 y < 

Program'^ ' Sook 
Tape Scr :pt § 
Intrinsically 1 
Programed book 


let 
.let 
let 


V-A 


5.1 
5.2 
5.3 
5.4 
5.5 
5.6 


Syndactic text 
Syndactic text 
Syndactic text 
Linear text 
Linear text 
Linear text 






V-B 

1 

\ 


5.7 
5.8 
5.9 
5.10 


Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text 
Syndactic tr/t 
Syndactic text 
Syndactic text 


V-C 
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TABLE 11 (Continued) 



Volume 
Number 


Segments 
Included 


Media 


Kemeci ai 
Media 


voiuntc 
Number 


! 

If T A 

, VI -A 


6. 1 
6.2 


Linear text 
Linear text 






: vi-B 


6.3 
6.4 


Linear text 
Syndactic text 


, VII-A 

1 

1 


7.1 
7.2 
7.3 
7.4 
7.S 


Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text 
Syndactic text 


VII-B 


Syndactic text 
Syndactic text 
Syndactic text 


VIl-C 


VIII -A 


8.1 
8.2 
8.3 


Linear text 
Linear text 
Linear text 






' VIII-B! 

1 


8.4 
8.5 

0 

0 . 0 


Syndactic text 
Syndactic text 
Syndactic text 


IX-A 1 


9.1 
9.2 


Audiotape/Panel book 
Audiotape/Panel book 


Syndactic text 
Syndactic text 


IX-B 




10.1 
10.2 


Audiotape/ Intrinsi - 
cally Programed 
booklet 

Audiotape/ Intrinsi- 
cally Programed 
booklet 


Tape Script 5 
Intrinsically 
Programed boo! 

Tape script § 
Intrinsically 
Programed bool 


X Script 
det 

:let 


XI 

i 


11.1 
11.2 
11.3 


Syndactic text 
Syndactic text 
Syndactic text 








12.1 
12.2 
12.3 
12.4 


Computer -Assisted 

Instruction 
Computer-Assisted 

Instruction 
Computer-Assisted 

Instruction 
Computer -Assisted 

Instruction 


Intrinsicallv 
Programed ^ jk 

Intrinsically | 
Programed book 


XI I -A 
let 
XII-B 
let 


Intrinsically |XII-C 
Programed booklet 


Intrinsically |XII-D 
Programed booklet 
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I 

I jFlGURE 15 

I 

I PROGRESS CHECK ANSWER AND REMEDIATION FORM 

I PART Eight SEGMENT IV 

I REMEDIATION TEXT Syndactic Text-Volume VIII-B 



ITEM 


ANSWER 


REMEDIATION REFERENCE 


] 


□ 




Summary 1 Pages 1-2 




d 




Summary 1 Page 1 


3 


□ 


Summary 1 Pages 3-4 


4 


□ 


Summary 1 Pages 3-4 


5 




a 




Summary 2 Pages 30-32 


A 

V 






m a _-Ll_[l__ M — — - _ O _ ^ C\ 

Summary 2 Page 28 


7 


□ 


Summary 2 Pages 30-32 


8 




c 




Summary 2 Page 30 


9 




a 




Summary 2 Page 32 


10 




d 




Summary 2 Page 33 


11 


□ 




12 










13 










14 










15 


□ 
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Individualization Within the Course 



As the original specifications for the project 
suggested, "there are many facets to the total question 
of individualization of instruction," It went on to 
suggest some of the methods whereby the process of 
individualization could be implemented in the course 
product. The first was that optional objectives, 
above and beyond the core curriculum, should be part 
of the total design. With respect to the media of 
instruction, the specifications suggested that the 
optimum medium would be selected after alternative 
presentations had been prepared and tested. Thirdly, 
individualization could be achieved by some branching 
procedures after pretests for prior competencies. 
Finally, individualization would be possible by making 
self-paced instruction the modus operandi for the 
instructional system, 

Westinghouse Learning Corporation was able to achieve 
significant strides in meeting these four requests for 
individualized operations. Indeed, individualization was 
such an intregal part of the entire development program, 
it was almost difficult to separate this concept out of 
the development effort and discuss it separately. 

Hopefully, some of the discussion to follow will 
illuminate one contractor's effort to achieve individualization 
with sufficient clarity and demonstrable value such that 




anyone interested in developing a self-paced individualized 
course will be willing to extend the effort, knowing the 
rewards that can accrue. 

Since this report deals with the final course design 
as the end product, a discussion of individualization for 
the final package is perhaps the most appropriate. In 
addition to the core curriculum of 59 instructional segments, 
seventeen depth core segments are available. As additional 
options for achievement within the course, these depth cores 
can be utilized to maximize individual interest and 
iiiotivntloii while Fulfillinj; the basic intention of integrating 
t'uiv oh.ioi-livos nl :t UtyMc- level of iiiulorstand ing . The 
deptli cores arc UiscusscU in more detail on page 208, 
aiul an example is provided in Appendix I. 

Additional enrichment is available through 27 
Structural Communication modules ^vhich are available in 
-Hh,,.. CH i.ulivhUu. 11 or .jrouj. mode. Although Jormal 
objectives, tested by specific criterion measures were 
not developed for these Structural Communication modules, 
their absence does not preclude their use as incentive to 
greater individual achievement. Indeed, the Structural 
Communication units, discussed in more detail on page 212, 
do not lend thcmsolvos to criterion referenced testing 
•since they can bo best described as dialogues on case 
studies ff londorr^Inp. 
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Thus, in addition to the 59 core segments of instruction, 
a total of 40 hours of additional materials were prepared 
with the express purpose of allowing optional course 
objectives to be attained. 

With respect to media, the final course product allows 
alternative paths for much of the instruction. In addition 
to fulfilling the requirement for selecting the optimum 
medium of instruction, the final course product allows 
alternative paths where the student's opportunity for 
individualization would be diminished by a reliance on 
hardware bound instruction. Additionally, the potential 
for individualization extends beyond criterion performance 
and takes student preference into account. 

Specifically, when the question arose as to the 
feasibility of having a large student population utilize 
the computer aided instruction facility at the Academic 
Computer Center, the instruction prepared for the CAI 
medium was recast into an alternate medium, an intrinsically 
programed text. This new format, surprisingly, proved 
as effective as the more sophisticated hardware medium 
version. Further details on the statistical results 
of the study are available in TR-6.15 and TR-6.18. 
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I Further, the potential requirement for supplying 

audio tape cassette recorders for a large number of 
students was instrumental in the conversion of all 
audio tape lectures into hard copy program materials. 
This move was also conditioned by the data which 
indicated that only a limited number of students, 
approximately 20%, showed a real interest in using the 
lecture tapes. Table 8 shows the full range of media 
options contained in the final course package. 

Individualization, through branching procedures, 
likewise is inhoront in the course. This project 
IH-oviUoil Lho I'irst, l.ir^-.e scale testing o£ a unique 
program materials format labeled a syndactic text. The 
syndactic text is a distinct individualized teaching 
device. Each unit of instruction is divided into three 
or more sub units which contain summary narratives of 
n.c lub(,uc< U.urtl cuucci.H. a bVioL- .luiz mid a detailed, 
programmed sequence for the content, and a final quiz. 
Both prior competencies and rapid learning rates are 
rewarded if the student successfully completes the 
summary quiz and he is allowed to bypass the detailed 
programmed sequence i'or that section. The syndactic text, 
'•"^''^'•'"""^«»i>ly oi.o „i; the more popular media in the 
course product, is available to the student in thirty 
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Additionally, branching as a technique to reward the 
able student, is available in other media throughout the 
course. The intrinsically programmed text, more 
generically known as scrambled text, allows the student 
to move quickly through material that he masters. The 
branching, conditioned only by incorrect responses, does 
not slow down the fast learner. This instrinsically 
programmed branching format is available in fourteen 
segments. 

Finally, the computer assisted instructional format 
on the IBM 1500 system needs no discussion on its 
branching and individualizing capability. In the final 
course package, four segments of instruction are available 
in this medium. 

The self-pacing requirement for the instructional 
system is realized through the logistical management of 
the course. Regular class meetings to attend lectures, 
as required by traditional courses, were abolished. The 
student reports to the Administrative Resource Room Csee 
Figure 16) to draw or return materials and equipment. An 
Individualized Study and Carrel Room located nearby Csee 
Figure 17) provides the student with space for private 
study or test taking. An instructor is available at all 
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Figure 17 
Individualized Study and Carrel Room 
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Carrel and study area with stools/chairs with. 15 stations. 

Approximate accommodations for 75-100 students, assuming a ratio 
of 3/A:l in "on demand" basis. 



Size: 23/24 
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tijnes for tutoring, therefore insuring assistance to 
the student on an individualized, as needed, basis. 
Further details on the role of the instructor in ful- 
filling his contribution to individualization is available in 
the Course Instructor Guide. Thus, the student can 
proceed through the entire course as quickly as he desires. 
The student is virtually without constraints to indivi- 
dualize his achievement. 
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Comparative Analyses of the Effectiveness of the Three 
Implementations 

In judging adequacy of segment performance, WLC es- 
tablished a criterion of 80-80, that is, S0% of the 
students must correctly answer 80?; of the progress check 
test items, if not on tht first attempt (before remediation), 
then on the second attempt Tafter remediation) or in 

individual tutoring sessions. Table i2 reports the 
percentage of students achieving the 801 criterion level 
before remediation on each of the three course implementations 
The mean percentages in this table show a vast improvement in 
overall segment performance from 801 - S7I in the first run, 
to 801 - 82% in the second run, and 801 - 94% in the final 
run. These data reveal that the requirement of the two 
revision cycles to reduce the need for remediation and 
tutoring was amply satisfied. In the third run there are 
only four segments (2.3, 2.6, 2.7, 3.4) which do not meet 
the 80 • 80 criterion before remediation. It should be 
pointed out that in three of these segments the number of 
test items was 8 or 9 rather than the customary 10. Actually 
only one segment (Segment 2.3) in the final run fell below 
the 80 - 80 criterion based on a 10-item progress check, and 
this was a marginal drop (77%). In the final run all 
students achieved at least 80% correct or better after 
remediation , and there were no instances in which tutoring 
was necessitated. Thus, in assessing the final run of the 
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Table 12 

Percentage of Students, by Segment, Attaining 
801 or Better on Progress Checks Before 
Remediation, Across Three Implementations 



Segment First Second Third 
Number Run Run Run 



1.1 


- 70.5 


1.2 


75.0 


2.1 


65.9 


2.2 


31.8 


2.3 


18.2 


2.4 


34.9 


2.5 


25.0 


2.6 


40.9 


2.7 


11.4 


2.8 


86.4 


2.9 


36.4 


3.1 


22.7 


3.2 


20.5 


3.3 


63.6 


3.4 


72.1 


3.5 


83.7 


4.1 


61.4 


4.2 


88.6 


4.3 


88.6 


4.4 


95.5 


4.5 


43.2 


4.6 


52.3 


4.7 


68.2 


5.1 


43.2 


5. 2 


22.7 


5.3 


52.3 


5.4 


25.0 


5.5 


-4.5 


5.6 


13.6 


5.7 


52.3 


5.8 


77.3 


5.9 


39.5 


5.10 


20.9 



91.1 


94.9 


80.0 


85.7 


86.7 


95.9 


85.7 


91.8 


35.7 


, 76.6 


69.0 


^ 88.8 


88.1 


97.9 


68.3 


69.4 


42.9 


73.5 


88.1 


93.9 


90.5 


96.9 


40.5 


80.4 


69.0 


86.6 


40.5 


100.0 


88.1 


73.5 


95.2 


90.8 


95.2 


97.9 


100.0 


96.8 


83.3 


94.8 


97.7 


98.9 


100.0 


96.9 


97.7 


97.9 


95.5 


95.9 


90.9 


90.8 


40.9 


86.7 


95.5 


97.9 


75.0 


93.9 


-/II. 1 


^96.9 


79.5 


87.8 


81.8 


96.9 


86.4 


93.8 


56.8 


93.8 


95.5 


91:8 
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Table 12 cont. 



Segment 


First 


Second 


Third 


Number 


Run 


Run 


Run 


6.1 


50 . 0 


95. 5 


95. 9 


6.2 


72.7 


81.8 


96.9 


6.3 


95. 5 


86.4 


94.9 


6.4 


79.5 


95.3 


95.9 


7.1 


18 .2 


69.8 


92.6 


7.2 


45.5 


90.7 


97.9 


7.3 


34.1 


69.8 


92.6 


7.4 


59.1 


60. 5 


93.7 


7.5 


43.2 


76.7 


98.9 


8.1 


70.5 


86.0 


93.7 


8 . 2 


42.9 


97.7 


97.9 


8.3 


88 . 6 


97 .7 


100.0 


8.4 


At o 

43 . 2 


97 .7 


100.0 


8 . 5 


60 . 5 


A A A 

88.4 , 


92.4 


8.6 


56.8 


79.1 


95.7 


9.1 


68.2 


93.0 


98.9 


fx o 


93 . 2 


93. 0 


98. 9 


10 . 1 


90 . 9 


97 . 7 


98.9 


in? 




07 7 


inn n 
1 u u . u 


11.1 


70.5 


74.4 


97.8 


11.2 


88.6 * 


100.0 


100.0 


11.3 


52.3 


95.3 


81.5 


12.1 


84.1 


97.7 


100.0 


12.2 


59.1 


74.4 


98.9 


12.3 


97.7 


93.0 


96.7 


12.4 


84.1 


67.4 


98.9 


Mean, 








All 








Segments 


56.8 


82.4 


93.5 
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course by percent-percent standards, 55 segments meet or 
greatly exceed the 80 - 80 criterion before remediation, 
and only 4 segments were below this level. After 
remediation, aU segments met the 80 - 80 criterion. 

Table 13 lists by segment the mean progress check 
performance before remediation for the three runs of the 
course. Once again the reader will note the improvement 
in course performance effected by each revision cycle, 
such that the mean performance increased by 8 and 6% 
respectively with each run, from 781 to 86% to 92%. In 
addition, it can be noted that in terms of overall mean 
performance there were only ten segments that showed a 
mean below 901. 

Time data across the three runs are reported in 
Table 14. The figures in the first and second runs 
represent the number of minutes spent on instructional 
materials as well as taking the progress check and performing 
remediation as needed. The time data for the third run are 
thus inflated by approximately ten minutes additional to 
take the progress check and remediate. When the mean time 
figure from the third run is made comparable to the time 
expenditures from the first two runs (by subtracting ten 
minutes from the average of 49 minutes) , it is apparent 
that the avferage amount of time per segment that the student 
invests in the course is lowest on the third run, while 
performance is highest on the third run, indeed, a very 

desirable combination. 24^ 
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TABLE 13 



Segment Progress Check Performance Across Tlnee Implementations 
Mean Perc-.itage Correct, Before Remediation 



Segment 


First 


Second 


Third 


Segment 


First 


Second 


Third 


Number 


Run 


Run 


Run 


Number 


Run 


Run 


Run 


1.1 


82 


87 


91 


6.1 


80 


90 


91 


1.2 


67 


84 


87 


6.2 


82 


85 


93 


2.1 


78 


88 


91 


6.3 


91 


88 


92 


2.2 


68 


86 


90 


6.4 


87 


87 


95 


2.3 


65 


70 


85 


7.1 


69 


81 


91 


2.4 


71 


78 


87 ■ 


7.2 


83 


87 


97 


2.5 


65 


90 


94 


7.3 


77 


80 


91 


2.6 


71 


82 


86 


7.4 


78 


78 


91 


2.7 


63 


72 


81 


7.5 • 


74 


83 


95 


2.8 


89 


86 


91 


8.1 


82 


87 


92 


2.9 


6S 


90 


91 


8.2 


81 


91 


93 


3.1 


66 


73 


84 


8.3 


92 


92 


96 


3.2 


66 


80 


88 


8.4 


76 


94 


95 


3.3 


75 


70 


95 


8.5 


77 


87 


93 


3.4 


79 


87 


88 


8.6 


77 ' 


87 


94 


3.5 


84 


93 


95 


9.1 


80 


91 


94 


4.1 


83 


92 


96 


9.2 


88 


91 


95 


4.2 


85 


95 


97 


10.1 


93 


95 


95 


4.3 


87 


84 


92 


10.2 


89 


^ 1 


95 


4.4 


92 


95 


96 


11.1 


80 


o2 


93 


4.5 


82 


95 


94 


11. 2 


91 


93 


97 


4.6 


73 


90 


93 


11.3 


83 


90 


92 


4.7 


77 


90 


93 


12.1 


87 


93 


97 


5.1 


78 


89 


88 


12.2 


85 


82 


94 


5.2 


70 


76 


85 


12.3 


94 


93 


96 


5.3 


74 


92 


93 


12.4 


85 


83 


94 


5.4 


69 


84 


90 










5.5 


59 


82 


94 










5.6 


70 


82 


92 


MEAN- 








5.7 


80 


83 


95 


ALL 








5.8 


90 


90 


93 


SEGMENTS 


78 


86 


92 


5.9 


78 


79 


90 










5 ..10 


66 


90 


94 











Table 14 

Mean Time- by Segment Across Three Implementations 



Segment Mean Time (No Minutes) 
Number 

First Run Second Run Third Run 



1 1 


Q2 

iJ id 


V u 


1 City 


1 2 




*T O 




2 . 1 


37 


51 


53 


2 . 2 


31 


48 




2.3 


31 


76 




2.4 


25 


49 


51 


2 . 5 


26 


58 


52 


2.6 


35 


68 ' 


68 


2.7 


30 


60 


67 


2.8 


34 


60 


62 


2.9 


55 


52 


48 


3.1 


37 


50 


42 


3.2 


25 


43 


42 


3.3 


32 


48 


41 


3.4 


27 


50 


44 


3.5 


34 


47 


44 


4.1 


39 


61 


65 


4.2 


43 


52 • 


43 


4.3 


41 


52 


60 


4.4 


56 


67 


63 


4.5 


50 


59 


53 


4.6 


41 


50 


49 


4.7 


43 


48 


41 


5.1 


57 


55 


52 


5.2 


59 


57 


51 


5.3 


39 


41 


48 


5.4 


51 


56 


51 


5.5 


52 


46 


50 


5.6 


40 


40 


35 


5.7 


31 


29 


38 


5.8 


39 


39 


37 


5.9 


55 


54 


44 


5.10 


35 


34 


32 
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Segment 
Number 



Table 14 (Continued) 
Mean Time (No Minutes) 

First Run Second Run Third Run 



6.1 


48 


67 


57 


6.2 


47 


63 


53 


6.3 


45 


60 


49 


6.4 


41 


44 


29 


7.1 


36 


41 


41 


7.2 


42 


46 


44 


7.3. 


38 


38 


28 


7.4 


42 


45 


41 


7.5 


39 


39 


34 


8.1 


59 


64 


72 


8.2 


60 


66 


63 


8 3 






ou 


8.4 


63 


44 


41 






qn 


A7 

H 1 


8.6 


56 


50 


40 


9.1 


42 


42 


35 


9.2 


40 


42 


37 


10.1 


52 


61 


56 


10.2 


45 


50 


48 


11.1 


31 


42 


38 


li.2 


30 


26 


31 


11.3 


34 


46 


43 


12.1 


68 


53 


60 


12.2 


71 


61 


64 


12.3 


38 


36 


48 


12.4 


41 


42 


54 


MEAN 


44 


51 


49 


NOTE: 


Data for the 


first and 


second 



time spent on taking the progress check 
and performing remediation. This extra 
time expenditure o£ approximately 10 
minutes is included in the time data for 
the third run. 
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I 

I 

■ It should be pointed out here that differences in 

I segment performance can be attributed to a multitude of 

factors such as differences in content, test items, media 
I and presentation forms and the personnel who developed the 

I materials. Thus when differences in segment performance 

arise, the difficulty lies in determining the causative 
I factors. The one factor which may most easily be isolated 

is that of the medium employed. To obtain some estimate of 
' the influence the medium itself has on segment performance, 

I one may average the results for material developed in each 

medium and contrast the averages. This has been done in 
I Table 15. 

^ One important qualification must be made in interpreting 

^ the results in Table 15. Although the materials have been 

grouped on the basis of media, the results should not be 
construed as evidence of the superiority or inferiority of 
one medium vis a vis another. These results do not reflect 
inherent qualities of the media as such, but are rather 
indications of the effectiveness of the materials which were 
developed for and presented in each medium. The reason for 
grouping and reporting results by media is to localize the 
variations in effectiveness of materials which may be 
attributable to teaching via different media. The results do 
not indicate comparisons of media made over identical content 
with identical test items, developed by the same writer, and 
employing identical presentation variables. 
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Tabic 15 

Progress Check Mean Performance (Bciore Remfediation) 
Across All Materials Developed in Each Medium, 
Second and Third Implementations 

V 



Media 


Second 
Run 


Third 
Run 


CAI Script/ IPB 


87 


96 


Audiotape Script/IPB 


93 


95 


CAI 


88 


94 


Audiotape/ IPB 


93 


94 


Linear Text 


88 


94 


Syndactic Text 


87 


92 


Audiotape/Panelbook 


82 


89 



The data for the third rt i in Table 2 was tabulated 
from the breakdown for media by segment given in Table 13. 
The breakdown within a segment in Table 13 indicates segments 
where a media option was available as was shown in Figure 2 
(page 108) . 

It can be noted that the mean performance within each 
medium increased from the 2nd to the 3rd run, and that 
performance is quite comparable across all media in the final 
installation run. 
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Table 16 

Mean Progress Chock' Performanco (Before Remediation) 
For Media By Segment for the Third Run 



Audiotape/Panelbook (AT/PB) , Syndactic Text (ST) 



Segment Total AT/PB ST 

Number Segment 

. Performance 



2.2 90 91 90 

2.3 85 84 85 

2.4 87 88 87 

2.5 94 91 95 



Mean (2.2-2.5) 89 89 89 



3.1 84 82 84 

3.2 88 82 89 

3.3 95 92 96 

3.4 88 79 89 



Mean (3.1-3.4) 89 84 89 



5.7 95 <85 96 

5.8 93 93 92 

5.9 90 88 90 

5.10 94 95 94 



Mean (5.7-5.1'^) 93 90 93 



7.1 91 87 91 

7.2 97 93 98 

7.3 91 91 91 

7.4 91 88 91 

7.5 95 92 95 



Mean (7.1-7.5) 93 90 93 
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Table 16 (Continued) 



Segment Total Segment AT/HPS §T 

Number Performance 



9.1 94 94 94 

9.2 95 92 96 



Mean (9.1-9.2) 95 93 95 



Mean for all AT/PB =89 
Mean for all ST = 92 



Syndactic Text 



Segment No; Segment Performance 

i.l 91 

2.1 91 

2.6 86 

2.7 81 

2.8 91 

2.9 91 
3.5 93 

5.1 88 

5.2 85 

5.3 93 

6.1 91 

6.2 93 

6.3 92 

6.4 95 

8.4 95 

8.5 93 

8.6 94 

11.1 93 

11.2 97 

11.3 92 



Mean 91 
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Table 16 (Continued) 



Audiotape/Intrinsically Programed Booklet (AT/IPB) 
Audiotape Script/Intrinsically Programed Booklet (ATS/IPB) 



Segment Total Segment AT/IPB ATS/IPB 
Number Performance 



4.4 


96 


95 


96 


4.5 


94 


93 


95 


4.6 


93 


93 


93 


4.7 


93 


91 


94 


Mean 


(4.4-4.7) 94 


93 


94 


10.1 


95 


96 


95 


10.2 


95 


93 


95 


Mean 


(10.1-10.2) 95 


95 


95 


Mean 


for all AT/IPB = 94 







Mean for all ATS/IPB = 95 



Computer Assisted Instruction- (CAI) 

CAI Script/Intrinsically Programed Booklet (CAIS/IPB) 



Segment Total Segment 

Number Performance CAI CAIS/IPB 



12.1 97 96 97 

12.2 94 95 94 

12.3 96 94 97 

12.4 94 92 95 

Mean 95 94 95 



Mean for all CAI = 94 
Mean for all CAIS/IPB = 96 
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Table 16 (Continuud) 



Linear Text 



Segment No. Total Segment Performance 

4.1 96 

4.2 97 

4.3 92 

5.4 90 

5.5 94 

5.6 92 

8.1 92 

8.2 93 

8.3 96 



Mean for all Linear Texts = 94 
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student Attitude and Conclusions 



The following pages are devoted to student attitude 
data as collected from tKe final installed course system* 
The actual questionnaire is shown in Table 17. The 
succeeding tables, summarizing the results, are almost 
self-evident and require little explanation to show that 
the final course system was well received by the students. 
This data on student attitude towards the course, in 
conjunction with performance data shown on previous 
pages proves that the processes employed to build the 
course yielded a valuable product, as well as a viable 
model* 
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TABLE 17 
STUDENT QUEST I ONNA I RE 



Volume number? 



a . 

b. 
c. 
d. 
e. 
f. 

fi. 
h. 



Audiotape/ I ntrinsically Programed 

Booklet (IPB) 

Audiotape Script/IPE 

Audiotape/Panelbook 

Computer Assisted Instruction fCAI) 

CM Script/IPB 

Discussion Booklet 

Linear Text 

Syndactic Text 



2 
2 
2 
2 
2 
2 
2 
2 



3 

3 
3 
3 
3 
3 
3 
3 



4 
4 
4 
4 
4 
4 
4 
4 



For the following questions circle the appropriate number. 

Above Below 

,., , . , . High Avg Avg Avg L 

Was the material interesting? — i~ "2 ~3 "a ~ 

Was the material difficult? 12 3 4 5 

Rate your approval of the way the 
material was presented (check only 
the media used in this Volume). 



If your answer to number 4 was "Below Avg." or "Low," identify 
the media and indicate why. 



5 
5 
5 
5 
5 
5 
5 
5 



Have you had any difficulties with regard to: 

a. Maintaining the minimum pace as indicated in the 
Course Activities Chart? 

b. Following procedures as outlined in the Student 
Guide? 

c. Obtaining instructional materials? 

d. Meeting with the instructor? 



Yes No 



If you answered "Yes" to any portion of number 6, please 
specify the area and the problem. 



If you have any additional comments to make with regard to the 
above questions or any other aspect of the course, please do so 
here. 



243 



I 

I TABLE 18 



STUDENT QUESTIONNAIRE DATA 

Question #2: Was the materia] interesting? 



Volume 
Number 


Segment 


Media 


# of 
Stdts. 


High 


Above 
Average 


Average 


Below 
Average 


Low 


I 


1.1/2 


SYN 


92 


10.3 


44.3 


41.2 


3.1 


1.0 


II-A 


2.1-5 


ATPB 


30 


16.7 


36.7 


40.0 


- 


3.0 


II-ABC 


2. i-9 


SYN 


60 


7.9 


46.0 


46.0 


- 


- 


II-D 


Z.6-9 


SYN 


91 


27.5^ 


49.5 


20.9 


2.2 


•* 


Ill-A 


3.1-5 


ATP'B 


13 


7.7 


30.8 


61.5 






III-BCA 


3.1-5 


SYN 


80 


3.8 


31.3 


55.0 


8.8 




IV-A 


4.1-4 




40 


5.0 


32.5 


52.5 


7.5 


2.5 


IV- B 
Tape 


~ J F — h 

4.5-7 


ATIPB 


26 


7.6 


23.0 


65.3 


- 


3.8 


IV- B 


4. S-7 


ATIPB 


61 


16.3 


36.0 


44.2 


1.6 


- 




5.1-6 


■ SYN 


92 


5.3 


21.5 


54.8 


15.0 


3.2 


V-B 


5.7-10 


ATPB 


11 




27.2 


45.4 




27.2 


v-c 


" 5.7-10 


■ iSYN 


83 




12.0 


78.3 


8.4 


9.6 


VI-AB 


6.1-4 


■"LP I 


91 


3.2 


28.2 


61. d 


■ 5.2 


3.2 


VI I- A 


7.1-5 


ATPB 


13 


15.3 


53.8 


23.0 


7.6 




VII-BC 


7.1-5 


SYN 


76 


CO 


31.3 


57.8 


4.8 




VIII-A 


8.1-3 


LP I 


91 


6.6 


24.4 


56.6 


7.7 


3.3 


VIII-B 


8.4-6 


sVn 


87 


4.6 


37.9 


54.0 


3.4 




IX-A 


d,i/2 


ATPB 


5 




lOO 








IX-B 


d.1/2 " 


■ SYN 


86 


3.5 


34.5 


5d.5 


1.1 


1.1 


X 

Tape 


10.1/2 


ATIPB 


11 


d.d 


45.4 


45.4 






X 

Script 


10.1/2 


ATIPB 


75 


2.6 


29.3 


64.0 


2.6 


1.3 


XI 


11.1-3 


SYN 


87 


4.6 


37.9 


49.4 


8.0 




Xll 


12.1-4 


CAI/IPB 


55 


11.0 


25.0 


55.7 ■ 


7.6 




XII 


12.1-4 


CAI 


27 


39.1 


39.1 


17.3 


4.3 
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TABLE 19 



STUDENT QUESTIONNAIRE DATA 

Question #3: Was the material difficult? 



Volume 
Number 


Segment 


Media 


# of 
Stdts. 


High 


Above 
Average 


Average 


Below 
Average 


Low 


I 


1.1/2 


SYN 


92 


- 


10.2 


62.2 


22.4 


S.l 


II-A 


2.1-5 


ATPB 


30 


- 


6.7 


83.3 


6.7 


- 


I I -ABC 


2.1-9 


SYN 


60 


1.6 


14.3 


74.6 


9.5 


- 


II-D 


2.6-9 


SYN 


91 


7.7 


39.6 


49.5 


5. 5 




Ill-A 


3.1-5 


ATPB 


13 






92.3 


7.7 


- 


III-BCA 


3.1-5 


SYN 


80 


1.3 


16.3 


77.5 


3.8 




IV-A 


4.1-4 


IPl 


40 


5.0 


5.0 


62.5 


17.5 


7.5 


IV-B 
Tape 


4.5-7 


ATIPB 


26 


- 


7.6 


73.0 


11.6 


7.6 


IV-B 
Script 


4.5-7 


ATIPB 


61 


1.6 


11.3 


80.3 


6.4 


- 


V-A 


5.1-6 


SYN 


92 


4.3 


18.2 


68.8 


7.5 




V-B 


5.7-10 


ATPB 


11 




18.1 


54. 5 


27.2 




V-C 


5.7-10 


SYN 


■83 




7.0 


78.8 


14.1 




VI -AB 


6.1-4 


LPI 


91 


1.0 


10.8 


82.6 


5.4 




VII-A 


7.1-5 


ATPB 


13 




30.7 


69.2 






VII-BC 


7.r-5" 


SYN 


78 




9.8 


82.7 


7.4 




VIII-A 


8.1-3 


LPl 


91 


2.2 


13.3 


76.6 


7.7 




VIII-B 


8.4-6 


SYN 


S7 




5.7 


78.1 


14.9 


1.1 


IX-A 


9.1/2 


ATPfi 


5 




20.0 


20.0 


60.0 




IX-B 


9.1/2 


SYN 


86 


1.1 


8.3 


61.9 


11.9 


1.1 


X 

Tape 


Id. 1/2 


ATIPB 


11 




27.2 


72.7 






X 

Script 


10.1/2 


"ATIPB 


75 


6.0 


7.6 


76.9 


11.4 


2.5 


XI 


11.1-3 


■"SYN 


87 


1.1 


8.0 


82.7 


8.0 




XII 


12.1-4 


CAl/lPB 






9.6 


82.6 


5.7 


1.9 


Xll 


12.1-4 


CAI 


27 




32.1 


64.2 


3.5 
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TABLE 20 



STUDENT QUESTIONNAIRE DATA 



Question #4: Rate your approval of the 

way the material was presented. 



Volume # of Above Below 

Number Segment Media Stdts. High Average Average Average Low 



I 


1-1/2 


SYN 


if L 


1 R 
± 0 • D 


5^ « 0 


Z3 . 7 


4.1 


JbOW 


rr-A 


2.1-5 


atpb 


30 


6.7 


3.3 


10. 0 




- 


Tl-ABC 


2 1-9 








4y • L 


53 • 3 


3 . 2 




11-D 




" SYN 


$1 


12.1 


46.2 


35.2 


6.6 


- 


III-A 


3 1-5 


ATlPR 




1 • 1 


1 0 • 4 


Z J . 1 






III-BCA 


5.1-5 


SVN 


ou 








1 . 3 




IV-A 


' 4.1-4 


LPI 






1 7 «; 






12 . u 


TV-B 
Tape 


4.5-7 


ATIt>B " 


26 


11.6 


19.2 


50.0 


11.6 

X X . \/ 


7 6 


IV-B 
Script 


4.5-7 


ATIPB 


61 


9.6 


16.3 


55.7 


11.3 


6.4 


V-A 


5.1-6 


SYN 


92 


6.8 


28.7 


62.0 


2.2 




V-B 


5.7-10 


ATPB 


11 


18.1 


72.7 


27.2 






V-C 


5.7-10 


SYN 


83 


6.2 


37.5 


61.6 


1.2 




VI-AB 


6.1-4 


LPI 


91 


6.5 


28.5 


62.6 


1.0 


1.0 


Vll-A 


7.1-5 


ATPB 


13 


7.6 


53.8 


30.7 


7.6 




Vll-BC 


7.1-5 


SYN 


78 


6.1 


29.6 


60.4 


2.4 


1.2 


VIII-A 


8.1-3 


■ LPl 


91 


1.1 


18.8 


63.3 


6.6 


8.8 


VlII-B 


8.4-6 


SYN 


87 


8.0 


35.6 


56.3 






IX-A 


9.1/2 


ATPB 


5 




80.0 


20.0 






IX-B 


9.1/2 


■ SYN 


86 


6.8 


38.6 


54 . 6 " 






X 

Tape 


10.1/2' 


ATIPB 


11 


9.0 


9.0 


63.6 


9.0 


9.0 


X 

Script 


10.1/2 ~ 


ATIPB 


75 


1.3 


19.6 


57.8 


10.5 


10.5 


XI 


11.1-3. 


SYN 


87 


8.0 


32.1 


59.7 






XI I 


12.1-4 


CAI/IPB 


55 


3.8 


5.6 


71.6 


16.7 


1.8 


XI 1 


"1:^.1-4 


CAl 


27 


55.1" 


31.0 


13.7 
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TABLE 21 



STUDENT QUESTIONNAIRE DATA 



Question #6a: Have you had any 
difficulties with 
regard to maintaining 
the minimum pace r.3 
indicated in the 
Course Activities Chart? 



Part 


Yes 


No 




I-V 


23.0 


78. 


0 


VI-XII 


3.0 


90. 


8 
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This chapter lias attempted to summarize the 
sequence of steps which Brought the media selection 
model from its conception to its final realization in 
the final course product. Its placement within the 
model is not entirely accurate, that is, while the basic 
design was one of the inital tasks of the project, 
quite obviously, the fabrication and production of 
materials (_the subject of chapter 5, immediately fol- 
lowinor) had to occur before any testing of the model 
was possible. Nonetheless, it seemed wise to utilize 
one chapter of this report for a comprehensive detailing 
of those steps with their appropriate examples which 
yielded the final course products. 

It is somewhat unfortunate that space is not 
available to include in even greater detail the many 
considerations, discussions, and "agonizing reappraisals" 
which centered on the problems of instruction? 1 
design. Anyone interested in more details may refer 
to Part I of this Final Report, TR 6.18, or the appro- 
priate documents listed in Appendix A. 

The costs involved in currently maintaining the 
course or in administrative overhead are essentially 



negligible. As reported in TR 6.17, Management Design , 
a Course Administrator can function as the interface 
between the students and one or more instructors per 
hundred students. The function of the Course Admin- 
istrator, located in facilities similar to those des- 
cribed in Figures 16 and 17, provides all clerical 
capabilities for course administration, thus freeing 
the instructor for the more suitable task of instruc- 
tion in whatever manner he sees fit. 
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V. DEVELOPMENT AND PRODUCTION 
OF INSTRUCTIONAL MATERTaIS 



Abstract 



This section summarizes the various 
competencies required for product 
development, the specifications for 
insuring uniform development activity 
in an R § D construct, and the manage- 
ment of the production effort. 
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v.. DEVELOPMENT AND PRODUCTION 



OF INSTRUCTIONAL MATERIALS 



This section will not attempt to detail all the 
procedures that are necessary for developing self-paced 
individualized instructional materials but will outline 
some of the relevant steps as they pertain to this model. 

Specifically, this chapter should specify those 
procedures that were employed which related to the 
specific tasks of creating materials that would be useful 
for research purposes as well as fulfilling the require- 
ments for course instruction. As such, it will concentrate 
on the requirements of the research model, levels and 
types of personnel, and scheduling procedures for con- 
tractual obligations. 
Personnel and Competencies 

Since materials development was the primary visible 
product of the project, it can be said that everyone 
wrote. There was virtually no administrative overhead - 
that is, every project member made direct contributions 
to the writing of instructional materials. Even the 
managers, nominally reviewers only, provided input as 
subject matter experts with pertinent examples, or other- 
wise employing their specialized knowledge for some par- 
ticular area of an instructional package. However, the 
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hulk of the development effort to prepare the instruc- 
tional materials was the responsibility- of the writers 
and editors. In the selection of personnel to be hired 
for these tasks, the decision was made that tke project 
could not afford to be a training ground for potential 
writers. Thus, all writers and editors were required 
to not only have previous background in educational 
technology and programmed instructional writing, but 
were screened by an examination which was devised to 
insure competencies in these areas. A copy of the test 
is included in Appendix J. 

In addition to an initial demonstration of com- 
petencies, the project personnel developed a range of 
abilities which evolved from their working with the 
various course products. Toward the end of the project, 
there was an effort to isolate those skills and competencies 
in a joint effort of Westinghouse Learning Corporation 
with a research team from Teaching Research, a division 
of the Oregon State System of Higher Education. A detailed 
statement of requisite skills for the various products 
were compiled by interviews with persons responsible for 
those products. The following list is a composite of 
those competencies which were elicited from the writers. 



Other competencies vhich relate to other pro- 
fessional tasks, such as design of research and evalu- 
ation plans, instructional design, and course models, 
are compiled in Appendix K. 
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CONTENT OUTLINES 



Product Tasks: 



Work with context analysis and writers to clarify 

course content 
Determine final format of content outline 
Relate all objectives to former course areas 
Decide on new weighting of parts--which areas should 

be expanded, which contracted 
Build content outline by topic which will work in 

the whole part traditional earlier course 

Assemble parts of content outline into booklet form 

Type content outline in standard format 

Remove redundant material from content outline 

Re-arrange material within content outline 

Revise content outline, based on comments from 

implementations supervisor and subjective 
opinion 

rn™LJ°f ^""^ outline to review for recommended changes 
Correlate/correspond content outline with course materials 
Ai^Ml^T^ verbiage between content outline and Sa^l^ials 
Add/delete examples where necessary. 
Determine parts that must be retyped or where can cut 

and paste 
Review for mechanical errors 

??n^rnn??-^''* outline based on current needs and changes 
Study outline and suggest deletions and changes 
Delete nonessential material from content outline 
Compare subcontractors recommendations against original 

° Items on contract outline 
Check to see if course outline includes all parts 

reg.t necessary 
Negotiate with (sponsor) for acceptance of each part 
w 1, ? stopped--still going on) 

Work toward concurrance/non-concurran5e on recommended 

Changes 

Assign writers tasks - corresponding/correlating content 
n«-h™° with course materials - based upon revisions 

Determine amount of time it would take a writer to revieS 

materials "^^^^^ content outline with course 
Determine a production schedule 
Interact with consultant about writing outline 
Submit content outline and recommended changes or 

modifications to implementation supervisor 

for review/approval 
Provide writers with clarification in relation to 

corresponding content outline and course 

matevial 

Double check writers work against revision specifications 

to see that specified changes were made as required ' 
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Apply for criteria: clear, concise, complete, correct, 

to revision process 
Ask other members o£ the staff for their judgment on 

technical topics 
Ask sponsor to resolve conflict over quality issues 
Negotiate with (sponsor) for emphasis, agreement on 

weighting of subjects of course 
Discuss in conferences the performance of staff, i. e. 

the writers ' 

Product Enablers : 

Knowledge of what midshipmen already know 
Knowledge of military circumstances through experience 
Knowledge through experience of midshipmen's life 
Knowledge of programmed instruction, principles and 
techniques 

Awareness of (sponsor) standards and context 
Knowledge of traditional previous course content 
Knowledge of (context) , methods and experiences 
Knowledge of mechanisms/operations (within content) 

to make recommendations on the content outline 
Knowledge of multimedia/mode used with a particular 

segment (knowledge of what syndactic, linear 

texts, etc. are) 
Ability to write programmed frames of instruction 
Ability to proof read for mechenical error 
Ability to proof read for consistency 
Thoroughness in work - ability to be thorough 
Ability to plan work - in terms of time and space 
Skill in writing clarity, etc. 

Ability to work with sponsor personnel in negotiating 
content of course outline and changes, etc. 

Ability to persevere in determining (sponsor's) desires 
thru negotiating 

Skill in determining what is non-essential in content 
outline 

Previous experience with content materials - helpful 
Sensitivity/awareness of end-use of the product - 

used at Academy by second class midshipmen 
Sensitivity to the content (user agency) - related to 

the above sensitivity 
Awareness of the effect of changes on the rest of the 

materials 

Awareness of the interrelationships of the materials 
Appreciation for time and what can be done in a time 
segment 

Awareness of personalities of staff you are working 
with 

Awareness of staff capabilities in relation to the job 
Awareness of the nature of the product and the steps/ 
pieces involved in producing that product 
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TBRMfNAL OBJECTIVES 
Product Tasks ; 

Analyze former course for areas of learning it covers 

results it expects ' 
Write objectives for each section working from new 

mediated course content 
Negotiate each objective with user for acceptance 
Build one terminal objective per major subdivision 

of every content outline 
Revise objectives on basis of added/deleted materials 

m content outline 
Delete objectives on basis of revised content outline 

Pr oduct i on P.nablcrs : 

Kiiowlcdj-.c of course content in terms of what/where 

material covered 
KnowlcdiH- of what user wants taught 
Kiiow!t.d,..(. of jiurpose of tennina] objectives enablinc 
«>l>.| «*t' I I v«':: 
ill i II wr i ( i n,; ol) j t. i vos 
••ikill at iiej'.oMai inj; acceptance or modification of 

.-acli tiM-miiial objective with usor 
A»i Ity lo use diplomacy in ivorkiiif, with writers 
A., , y o correlate material with terminal objective 
Ah. I Ity o read caro fully - to see tliat the objectives 
;..u material correlate and correspond "^'"'"''''''''^ 
Sonsitivxty to the interrelationships ii; dcrivinq 
<n.-'l'lin,; objectives from terminal objectives 

i;NAiii. VNf'; uH.iiiji'ri vi;;: 

['rodiifi la six-. : 

■ 

.■••..I, ;.;r;:r,,:'',';,,',;;L;;';?v:;''- • j'"-'"" 

I'rodiii- t llnabj ers : 

KnowLedge of what (sponsor) wants 

f.kill n, i„ ii.i„,. olijiviivo:; 

ol'i«f?«f jo? "ff " modification of 

obiLuvL 'fro; ' icainal 'objoctivos 
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SYNDACTIC TEXT MODULES 



Product Tasks ; 

List the topics to be covered iji text unit 

Read material Cprimarily texts] in the office - in relation 

to the subject matter of a particular syndactic 

text, (literature review) 
Collect textbook material to read/review relative 

to content 

Consider level of material in terms of audience 
(midshipmen) 

Determine which material (from textbook review) will 

be used to support a particular point relative 

to subject matter 
Specify and/or create graphic materials to be included 

in syndactic texts 
Number the topics in sequence of presentation 
Correlate content outline with material for syndactic 

text - 

Determine number of summaries to be included within 

each syndactic text - there are major parts/ 

sections of the text 
Establish guidelines/steps to follow in writing a 

syndactic text format 
Use progress checks as a reference in writing material 
Think of original/sensible ways of presenting and 

testing information/content 
Write pre-quiz items to cover basic points in the 

syndactic text 
Write summary/ overview part of the text 
Write frames - an expansion and check of points covered 

in original summary 
Scramble/rearrange postquiz items and answer3 - same as" 

prequiz items after summary 
Obtain the master material to be summarized 
Obtain the terminal objective for this unit 
Rewrite the topic titles to fit into space required 
Write new summary of segment which needs revising 

accommodating criticisms of information 
Write new linear frames to go with summary 
Write new summary quiz to match new summary in content 
Produce the interview form for interviewing midshipmen 
Type luaterial for camera copy readiness 
Interview midshipmen about effect of product 
Use an interview form during the interview of the midshipmen 
Utilize revision related comments from the user agency 

as to major points to cover in syndactic text 
Modify content outline on basis of readings (literature 

review relative to syndactic text) 
Review syndactic text in relation to content outline to 

make sure every point in the outline has been 

treated/acknowledged 
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Check to see that progress checks correspond until 

material is actually taught 
Check that terminal objectives and enabling objectives 

for the segment were covered 
Edit entire text of segment for more types 
Improve phraseology throughout entire segment of course 
Read the segment which has been sent back for revision 

as it stands, and read comments which indicated 

need for revision 
Check product against criteria 
Prepare recommendations for modification 

Make recommendations for changes in instructional material 
Proof read material - typists/production/writers/editor 
Modify/incorporate recommended changes from user agency 
and editor 

Discuss instructional examples until subject matter expert 

for correspondence with reality of military experience 
Send over pieces of the text for the user agency to review 

and make suggestions and/or corrections 
Review materials sent by the contractor to the user 
Assist other writers in writing a syndactic text 
Interact with sponsor as to nature of the material in 

syndactic text 
Provide all writers with a set of format guideline^ 
Hold committee meeting to review depth, core materials 

(those so specific it required more specific expertise 

Product Enablers ; 

Knowledge of content outline so that proper objectives 

are covered in revised form 
Knowledge of function of TO's and EO's (terminal objective- 
enabling objective) 
Knowledge of syndactic course structure 
Knowledge of leadership course content 
Knowledge of the media being used for each section 
Knowledge of jargon used by midshipmen 
Knowledge of military practices and terms used 
Knowledge of who to ask if knowledge not in head 
Ability to write concisely 

Knowledge of objectives in course and specific unit 
Knowledge of contents of course and specific unit 
Knowledge of rudiments of writing (instructional materials) 
Knowledge of Navy jargon 

Superficial/introductory knowledge of content area of 
syndactic text 

Knowledge of how to write test items - primarily multiple 

choice type items 
Ability to write a paragraph - related to the summary 
Skill in ways of presenting and testing material - frame 

writing styles 
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Ability to work with data - read data sheets to pin point 

test items that might need revision 
Ability to specify graphic illustrations/charts to a 

accompany text 
Ability to draw the illustrations .figures to accompany 

text 
Ability to type 

Ability to look at index and determine where irost 
useful information relative to a partici lar 
topic would be found in the publication 

Ability to maintain a continuity or flow in the sequence 
of frames that are written in a syndactic text 

Ability to logically approach a subject or topic 

Skill in writing 

Ability to be persistent and painstaking in reviewing 
materials 

Experience in teaching subject matter 

Ability in writing in order to write readable examples, 

portions of text 
Sensitivity to the specific characteristics of target 

students 

Sensitivity to needs of students beyond class circumstances 
Sensitivity in the balance in writing a paragraph - 

points in paragraph as specified by topic sentence - 

have covered 



VIDEOTAPE MODULES 
Product Tasks : 

Determine where training aids (visuals) might be needed 

in conjunction with the presentation 
Suggest corrections in videotape in both format and 

content 

Suggest changes in verbal script 
Edit scripts for accuracy 

Remake video visuals for different (3) response mode 

(as against the 20 response mode) 
Make sure equipment hardware and software are compatible 
Plan content to suit medium in interesting manner 
Write/prepare script 

Make copy of first script for approval 
Transfer script to teleprompter 

Organize hundreds of slides for visuals (and panelbooks) 
Transfer video visuals to 35 mm slides 
Duplicate video visuals for panelbook (for audiotape) 
Prepare visuals for appropriate, superimposition 
Select reader - someone who knows or is professional 
reader 

Hiring of subcontractors to care for di'ection production 
Purchase video tape 
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Hiring of talent for performii g lecture material 
Select talent personality to suit intended presentation 
realistically 

Contract local commercial TV station to use facilities 
Contract another facility to transfer to 1 inch video 
tape 

Record the video tape 

Edit video tapes for errors of commission and demission 
Detect errors in video tape - both technical and in terms 
of jargon 

Provide comments as to quality of video tapes 

Insure that script was followed by the speaker who was 

cutting the tape 
Edit video tapes 

Edit errors, electronically, as made 

Rewrite many scripts for better video contact with viewer 
Eliminate unnecessary big words - be concise 
Get tapes duplicated 

Product Enablers; 

Knowledge of audio-visual hardware for media 
Knowledge of production techniques relevant to individual 
media 

Knowledge of audio and print writing technique 
Knowledge of capabilities and limitations of video as 

electronic medium 
Knowledge of how to run video tape machine 
Knowledge of user relevant terminology 
Knowledge of user procedures and policy 
Ability in designing and coordinating video and visual 

elements of presentations 
Coordinating all the people involved at one time or in 

sequence 

Ability to read as if he knows material (narrator) 

Ability to adapt script, etc., to each medium 

Ability to fit everything into scope of medium being used 

Ability to make creative decisions* 

Ability to write for both audio and print 

Use of concise, simple language - not educationese 

AUDIOTAPE MODULES 
Product Tasks : 

Transfer sound track from video to audiotape 

Develop scripts complete with instructions to reader re: 

pausing, emphasis, etc. 
Get panel book made up to correspond with video visuals 
Duplicate tapes in house 



Product Enablers : 

Knowledge of audio -visual hardware for me^^ia 
Knowledge of production techniques relevant to 

individual parts 
Knowledge of audio and print writing technique 
Ability to read as if he knows material (narrator) 
Ability to adapt script, etc., to each medium 
Ability to fit everything into scope of medium 

being used 
Ability to make creative decisions 
Ability to write both for audio and print 
Use concise, simple language - not educationese 

COMPUTER AIDED INSTRUCTION fCAn~MODULES" 
Product Tasks 

Duplicate material for use on other media, as 

comparison and to replace 
Deleted some irrelevant materials from instructional 

program 

Product Enablers : 

Educational technology as taught by Skinner, et. al. 
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LINEAR PROGRAMED MODULES 



Product Tasks : 

Review reference materials - textbooks relative to 

content material of the texts 
Correlate content outline with materials in the 

linear texts 

Incorporate material fror^ references in*-.o writing 

of the frames 
Select examples, usually user relevant, vo support 

material 
Write/create frames 

Confer with subject matter experts as to ai propriateness 

of examples to be included in the lii ear texts 
Provide implementations, supervise alternat ve ways 

of expressing a particular point in the material 
hollow directives from the user agency for 

specifications in revision of the materials 
Participate in decision of final format/style - input 

and information 
Interact with implementation supervisor for fu7ther 

clarity/specifications in revising linear 

materials 
Type material in camera-ready form 
Rewrite frames- -content within lin^^ar text 
Add and/or delete frames 

Revise style of f rames--bring paragraphs together; 

rearrange order of frames, add examples 

Review/proof-real final materials of linear text 

Revise sentence-structure/grammar- -does not change the 
meaning j 

Product Enablers ; 

Ability to revise frames relative to feedback from 

testing 
Ability t write a frame 

Skill of tuct - particularly in dealing with the 
user agency 

Ability to persuade or convince of your viewpoint 
INTRINSICALLY PROGRAMED MODULES 
Product Tasks : 

Test material With sampling o£ three to five students 

at contractor site 
Edit, revise, "patch" copy in rush to meet deadline 
Apply references from faculty and subject matter to 

experts 'for revision of textual content 
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Rewrite sections based on deficient student 

(statistically) performance 
Patch "errors" shown by students making incorrect 

responses 

Retype copy marked up for sendim^ to printers 
Bind and staple copies of programed texts 
Textual specifications given and explained to 

subcontractor 
Examine and critique text (copy received from 

subcontractor) 
Decide to redesign sequencing for programed text 
Prepare schematic for producing modified instructional 

sequencing that meets standards of programing 
Tram other writers to produce optimally programed 

intrinsic programed texts 

Product Enablers : 

Knowledge of "process of Preparing Valid Effective 
Learning" 

Know|cc(lj;c. of ^-ood programing frames from bad frames 
Ahility in writ inn suitable sequences for intrinsic 
prof. raiiM'.l I c;\ rii i lU' 

ni ii,(rr,„H.i inn "Proparing Valid liffcctivc 
I. earn 111)'" 

Ability in producing schematics for scrambling 
proj'.ramod so<iuonccs ^ 
Hxpcrioncc as military instructor 

""'"L'Srll^i^ " ^"""-^ contradictions 

Ability to think like a student 
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Development.! Guidelines 

There was no single HanJbook wliich. compiled 
all specifications for tLe developers of instructional 
materials. Tliere were, however, some documents which 
provided guidelines to the developers which are included 
in the appendix. The document in Appendix L represents 
one type. It. is a brief summary of programmed writing 
procedures as they pertained to the specific products 
under development. 

The second document is the Manual of Style, a 
portion of which is shown in Appendix M. A third input 
to the writers which provided guidelines for the research 
activities was found in two documents, TR 6.3a, titled 
Research and Evaluation Plans , and TR 6.4a, Instructional 
Presentati on Design . These two documents specified which 
segments and modules of instruction would be developed 
with varying presentation characteristics. An extract 
of the presentation design is shown in Figure 12 in 
Chapter 4, page 162. In capsule format, it presents the 
variety of presenilation forms and the specifications for 
each learning module. A detailed explanation of the 
meaning and rationale for each presentation is presented 
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in Part I of the final report, TR 6.18, Summary and 
Conclusions Relating to Research and Evaluation of a 
Multi-Medi a Instructional System . An extract from TR 6.3a, 
which provided more specific guidelines to the writers 
is presented in Appendix N. 

Guidelines for content were the Content Outline, and 
Terminal and Enabling Objectives, samples of which are 
sLown in Appendix 0. Further guidelines for content 
were the various source materials or bibliograp'.y which 
provided an input for the original definition of the 
Content Outline. A samr-le of the bibliography used is 
presented in Appendix P. 

All these guidelines combined to give the instruc- 
tional writers and editors specific and detailed speci- 
fications for their development work. 
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Management of Development and Production 



Since the development effort would req ire the 
drafting, writing, and production of over two hundred 
original modules of instruction, the problems of control 
and scheduling were exceedingly complex. The basic 
procedure for each module of instruction was to generate 
the primary module which contained the basic instruc- 
tional content. On approval of that content, the sub- 
sidiary modales incorporating the variations on presen- 
tation design characteristics were then generated. Figure 
18 represents the development effort of the primary 
module of instruction. It is a matrix showing the time 
sequence of each stage in the development effort prior 
to submittal. 

Another example of the control procedures is 
shown in Figure 19, which indicates the various stages 
of development and production and the use of the MS 
(manuscript) control jacket which contained all pertinent 
data' on .each manuscript in addition to the draft as it 
moved through several departments. 

A sample of the delivery schedule is shown in 
Table 22, for the sixty segments of instruction as in all 
subsidiary modules. It is offered here to show the complexity 
o£ the schedule required ".o maintain smooth operations. 
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Primary Module Development 



D 





Legend 

W Writer 

S SME Review 

E Editor 

M Management 

T Typing 

P Proof Reader 



W 
S 
A 
3 



PC Z 



M 
4 



N 

Z 

PR 
DC 



Navy Review 
Zerox 
Printer 

Document Controller 
PC Page Checker 
A Artist 



2 Writer/ 
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1 Writer w/ artist § SME must produce a smooth- rough of tcict ^ n^xccj 
SME review of smooth text 3 Writer/artist/SME review ti make USNA recom- 
mended corrections 4 Final management quality control revifcw prior to 
submission 
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General Considerations for Developnient and Production 



Personnel requirements to achieve the goals for 
a large sca.le production effort are dependent upon two 
factors; the first is the amount of work to be done, the 
second is the time frame. If there are no deadlines 
to be met; that is, the material does not have to be 
ready for student use on a given day, then a handful 
of writers can produce any given quantity of materials • 
When deadlines have to be met, the staffing requirement 
is increased and the necessity for control and contin- 
uity becomes greater. It should be obvious from the 
schedules shown in Table 22, where finished products 
were delivered at the rate of two to four segments a 
week, that the manpower loading level was quite high. 

As implied from the charts and figures presented 
in this chapter, there was a great effort made /o insure 
continuity and uniformity of materials. All manuscripts 
received several reviews at varying levels. There was 
a great deal of interaction and discussior* among all 
levels of the staff, and between the writers and subject 
matter experts. Thus, despite the fact that oyer twenty 
five writers were employed in the development effort, there 
is an amazing consistency of style, format, and quality. 
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This chapter has attempted to delineate both 
those competencies required for product development 
and the guidelines which allowed the writers to produce 
acceptable and well-performing materials for a variety 
of fori,'ats. Quite obviously, there were difficulties 
in producing such a large quantity along lines that 
such a project demanded. Generally, however, they were 
resolved through more attention to comraunica.tion between 
various staff levels. 

Anyone who undertakes product development often- 
times asks the question if it is more cost effective 
to produce materials in-house or to sub- contract to 
consultants. or incorporated materials development groups 
It was the experience of this contract that all develop- 
ment work had to be in-house products. In other words, 
although consultants in various subject matter disci- 
plines were used to generate some of the materials 
to be used, it became necessary to maintain a large 
in-house staff for rewrite and editing. Thus, although 
drafts were prepared by tie non-permanent staff, the 
responsibility for producing the final instructional 



material fell to a team of three in-Iiouse personnel - 
a writer, editor, and subject matter expert, 

THe importance of the operations discussed in 
this chapter again suggest the pressing need for extra- 
ordinarily^ competent personnel at both a technical, 
editorial, and management level. 
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VI. DEVELOPING AND 



VALXDATING TEST ITEMS 



Abstract 



This section details the procedures 
used to develop various test instru- 
ments for the course. It specifies 
the guidelines to the writers as 
well as the criteria employed for 
the revision of test items. 
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VI. DEVELOPING AND VALiJATING TEST ITEMS 
f Introduction 

-: Th-ere were two types o£ test items developed 

J for th-e multimedia Leadership course. One type was 

criterion referenced for evaluating and validating 
I instructional materials in their ability to 

f ^^^"g tJ^e student to achieve the objectives of 

' I course. The performance level for these 

;^ ^ objectives was set at 80/80 - that is, eighty percent 

I of the students were to achieve 801 of the objectives 

? I "^^"g criterion referenced test items. The second 

i research test item which was developed to 

f 1 ^^^^ experimental hypotheses of the project. Since 

f I hypotheses were aimed at establishing the effects 

I ' °^ different instructional treatments rather than 

I j establishing proof of high criterion performance, they 

: "'^'"^'^ ^® loosely classified as norm referenced test 

! I ^^^^s- "ence the distribution of scores on these 

I J research tests would conform more to the bell shape 

J ' not be clustered at a specific level, as 

i j *^it^ the criterion referenced items. 

t Criterion Referenced Items 

i i Discussing development of criterion referenced 

: test items can best be done by examining their relation- 

i ®^^P to the objectives for the course. The ideal 

; objective, in stating the understanding or performance 

• to be demonstrated, specifies the conditions of the 

behavioral measure. Hence the structure and some key 
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components of the test item are created in the objective. 
Additionally, the sequencing and hierarchies of the 
collective objectives specify the types of learning 
that the criterion test items will evaluate. The type 
of learning, especially as it relates to other competencies, 
is not always explicit in the objective statement. Thus 
the criterion test item is dependent on the component 
specification of the objective for its structure, and 
is dependent the objective's placement in the hierarchy 
of objectives for an accurate interpretation of 
compotoncics alon,. the scale of types of learning. 

TIk-.o uli;;orv.il ion;; aro i 1 I usi ratoj with the 
'•'ll-w.un .x.-n,.,,h.... i-.-o,,, U.o writers spcci f lent Jon used 
i" H.is projocl. Assume that for purposes of uniformity, 
the vnrious theoretical constructs of learning types were 
reduced to four: (1) Definition and Identification, 

and i>roble,„ Identification, and (4) Problem Solving. 
Our goal was to set specification for both the statement 
of objectives and the general construction of criterion 
referenced test items. The following guidelines were 
therefore developed and presented to the writers as 
specifications for criterion reference items. 
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Specifications for Criterion Referenced Items 

Type 1 - Definition - Identification 

Given the instruction to identify the correct purpose 

of (definition of/description of/use of) concept X, 

the M will select from several choices the correct 

purpose of (definition of/description of/use of) concept X. 

1 . General Type 1 

a. Given the instruction to define concept X, 

b. The M will select from several choices a 
definition of X 

c. Similar to the following: "X..." 
(NOTE: The third part is optional.) 

2. example of Type 1 

a. When given the instruction to define "acquisition," 

b. The M will select from several choices the 
corrept definition. 

3. Example of Type 1 

a. Given several choices, the M will select the 
correct definition of acquisition. 

4. Example of Type 1 

a. When asked to define "attention," 

b. The M will select from several choices a statement 

c. (which indicates that attention is the selection 
of specific stimulus elements) . 
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Type 2 learning is a composite of those intermediate 
skills which many deem necessary before passing on to 
higher and more complex abstract operations. Hence the 
following: 

Type 2 - Discrimination - Comparison 
Given the instruction to evaluate the relationship 
between (defining attributas of/contrast between/ 
comparison among) classes X, Y, Z...N, the M will be 
able to select from several choices the paragraph 
which illustrates (describes/differentiates/identifies) 
the relationship between (defining attributes of/ 
contrast between/comparison among) classes X, Y, Z...N. 

1 . General Type 2 

a. When asked to evaluate the relationship 
among X, Y, . . .N, 

b. The M will select from several choices 
the paragraph which describes this 
relationship. 

2. Example of Type 2 

a. When asked to evaluate the difference 
between retroactive and proactive 
inhibition, 

b. The M will select from several choices the 
paragraph which describes this difference. 



"6 291 



Type 3 corresponds to the first entry into higher 
order analysis and a complete break from simple or 
lower order intellectual activities. Hence: 
Type 3 - Generalization - Problem Identification 

1. Deductive 

a. Given examples of X, the M will be able 
to select from several choices the example 
which illustrates principle Y. 

2. Inductive 

a. Given an example of X, the M will be able 
to select from several choices which 
principle (X, Y, ... or N) is shown 
(exemplified/demonstrated) by the example. 

3. General Type 3 

a. Given examples of an X, 

b. The M will be able to select from several 
choices the example which illustrates 
principle Y. 

4. Example of Type 3 

a. When asked to compare several versions of 
the same communication, 

b. The M will select from several choices the 
version which clearly links the subordinate's 
role to the overall objective of Naval 
operations. 
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There is general agreement that problem solving, using 
complex and almost simultaneous operations o£ analysis 
and synthesis represents the highest order o£ learning 
activity. Thus: 
Type 4 - Problem Solving 

When asked to evaluate a situation which is an example 
of class X, the M will select from several choices the 
correct solution (approach/method/resolution of/ 
reaction to) the situation, using principle Y. 

1 . General Type 4 

a. Given a problem situation which is an example 
of class X, 

b. The M will select from several choices the 
correct approach to the situation, 

c. Using principle Y. 

(NOTE: The third part may be omitted if the 
objective is unambiguous. In practi-ee, the 
third part is usually not given to the student.) 

2. Example of Type 4 

a. When asked to evaluate a situation in which 
there is an apparent failure in communication, 

b. The M will select from several choices the 
description which indicates the appropriate 
action of a leader, 

c. Who assumes responsibility for the failure. 
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Further guidance to the test item developers was the 
content outline, which is something of a misnomer for the 
thirteen volumes of substantive and detailed explication 
of the specialized knowledge germaine to Leadership, 
Psychology and Management. The primary contribution of 
the content outline as guidance was the assurance of 
content validity for the items. 

The personnel, outside the main development site, who 
prepared test items were for the most part, experienced 
behavioral scientists whose work was later reviewed for 
military appropriateness. 

Other Guidelin es 

In house, the test items were developed by two man 

teams which consisted of an educational technology 

specialist and a subject matter specialist who had 

primary experience as a naval officer and secondary 

expertise in at least one of the subject disciplines. 

There are few hard and absolute rules for test item 

generation since literary style is often as great a 

factor in successful test item writing as behavioral 

expertise. One piece of guidance circulated to 

writers and abstracted below suggests this point when 

it cautions that the test item writer have: 

.♦.an insight into backgrounds, abilities, 
and particularly the mental processes of the 
subjects who are to take the tests a facility 
in the clear and economical use of the language, 
and above all, an essential willingness to 
devote the time and energy necessary for the task... 
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Some persons I once they have overcome their 
initial blocking, can produce items steadily day 
after day... Others, however, are never satisfied 
to reduce an idea to a reasonably abbreviated 
statement with the addition of several alternative 
answers. They can always think of myriad details 
about which one might wish to have information 
before committing himself to an answer^ 

On the relative objectives of the test items and 

their relationship to the objectives of the course, the 

following advice was offered. 

The function of the progress checks is to 
insure that minimal acceptable levels of 
performance are attained by all students in 
certain essential behavioral objectives. These 
objectives are essential either because they form 
part of the behavioral outcome to be shaped by 
the course, or becuase they are prerequisites 
to some later stage of training. Clearly, the 
progress checks should be at a minimal level of 
difficulty and closely keyed to the specific 
behavioral objectives around which the 
instructional segment is designed. If the item 
is constructed with content validity at a 
minimally acceptable level of performance 
established on an a priori basis, it may then 
serve as a criterion for evaluating the 
adequacy of the instructional segment. If 
subjects miss the item, then the instruction 
is to blame, not the student. Thus, if part 
of an instructional segment is to teach the 
student to discriminate the presence or 
absence of a particular variable in a social 
situation, an item explicitly testing that 
discrimination would be included in a progress 
check, using an exemplary situation similar to 
those used in instruction and which "all 
reasonable men" could agree should be correctly 
answered if the goals of the course are to be 
achieved* 

Specification for Research Tes t Items 

— 9 

The development of research test items was 
predicated on the belief that criterion referenced 
items and tests (tied to specific objectives) would 
not be sufficient in measuring either the differences 




in instructional treatments or the larger intellectual 

cognitions that would derive from a full semester of 

intensive study across a variety of subject disciplines. 

As posited in an early position paper on the posttests 

[the vehicle for research test items] . 

Given that a student attains the minimal 
levels of performance defined by the progress 
checks, we may still expect the students to 
have achieved widely varying levels of ability 
to generalize to novel situations, to resist 
interference by similar material presented 
earlier or later, to integrate new ideas into 
larger conceptual schemes with broad behavioral 
implications, i„nd to deal with more subtle 
and complex examples. The posttests must be 
designed to pick up the differences in this 
partial or generalized behavior if we are to 
determine what kinds of behavior change are 
produced in those segments of the domain. 

In the specifications that were detailed for these 
research post- test items the following are of interest. 
1. Relationship of items to content: Cumulative 
Posttest (CPT) items will be prepared on the basis of 
content outlines and the content of "common" instructional 
materials (i.e., materials presented in each of the 
parallel experimental modules). 

Each CPT item must have an identifiable relationship 
to one or more general and/or specific headings of the 
content outlines; references to content outline headings 
will be noted on. each test item submission form. 

Each item will be related to the content of as many 
different segments making up the Cumulative Posttest Unit 
as is feasible and desirable in relation to other 



^" 296 



specifications detailed below. The entire set of items 
should represent an equitable distribution of content 
among segments of the Unit in relation to the relative 
amount of content covered by the outlines and the 
materials of each segment. 

3. Item difficulty: CPT items will possess high 
discriminative power; WLC will endeavor to construct 
items of relatively high initial difficulty and 
sensitivity to gain, using the general guidelines that 
not more than 40% of the students should answer the 
item correctly on a pre-test, nor less than 501 of the 
students should answer the item correctly on a Posttest. 
Actual indices of item difficulty will be determined 
in the validation testing program at the USNA, as a 
basis for subsequent item revision. 
5. Types of items: CPTs will include items which 
represent "acquisition of knowledge" (Type I items) 
and "application of knowledge" (Type II items). 

Type I items will measure acquisition of specific 
factual information covered in one or more segments of 
a Unit. Acceptable forms of Type I items include 
definition- identification, discrimination-comparison, 
multiple discrimination, and concept identification 
question. 

Type II items will measure application of concepts 
and principles', covered in one or more segments of a Unit. 
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Acceptable forms of Type II items include problem 
identification (selection of the correct or most 
appropriate example or illustration of a given 
concept or principle) and problem solving (selection 
of the correct or most appropriate solution of a 
Kivcn problem). Hach CPT will contain approximately 
c(nial mimbcrs of Type I and Type II test items. 
I'rocodurcs Cor U c-vL.slon oT Test items 

Tlic ftcncral criterion for revision of cither 
test items or materials was the .S()"„ difficulty level 
of ail i 1,-111. IV)mmi i(,-iii;. ho ! ,nv 1 ho «()'• Wvcl, 1 iic 

K.-m. ..hj... ( ,v.-, .iml Cu.HvuL WcTo rovioWod with 

••aspect to tho accuracy of their interrelationship. 
WluMV discvpancics occurred, subjective judgment 
determined if the test item or the materials should 
be changed in order to achieve a closer correspondence 
to the objective. Where no discrepancy appeared, the 
following procedures were implemented: 

The distribution of responses to each distractor 
wa:; studied in relation to the materials in order to 
determine a) if the distractors were in fact correct, 
b) if the distractors were incorrect but not presented 

i"-.-r...,. oi. .) ir ,,,,,,, 
w.-i:. iM.l .Mii(.ha;;i/,rd a-, curviwi . 

" -'•-n..i...H.„ .h. ....u-on.,,; a.s for 
the correct response by incrensin, the similnrity of 
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examples in the materials and the example used in the 
test item, or vice versa* Where incorrect distractors 
were partially or actually correct, subject matter 
experts made decisions on revising the content outline 
or material. 

When available, the discrimination index of each 
item was used in conjunction with the difficulty level 
in deciding revision. Although the difficulty level 
was weighed more heavily in revision, all negatively 
discriminating items, regardless of difficulty indicated 
need for revision. Since negative discrimination 
indicated that students who scored high or all other 
items were failing an item, the materials were reviewed 
to see if in fact the correct answer was the only correct 
response or if there was another distractor which was 
also correct* If the difficulty level was 90% or above, 
a discrimination index was not considered, since only 
two or three students could cause the discrimination 
index to be negative. However, if the difficulty ranged 
between 80 and 90 and the discrimination was negative, 
•che need for revision to either test item or materials 
was clear. 

Althouj^Ji tho d i sc r i m i na I i on indiecf; v^on^ not 
nviiilahli^ lintil al'liu* many iotA i U-nis atitl ma ( 4M' i a I 
were revised, all progress checks were checked to 
determine iT items that were ne^iatively <l i scrimiiint inji 
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or significantly posiiively il i .sc r im i n:j t i nj.'. should bo rovlsod 
under tlic dirflculty rule. General }<uldolincs Tor use 
of I>otli diri'i cully and discrimination in revision arc 
J', ivon below. 



REVISION 


DECISIONS BASED 


OK PROt. . 


CHECK 


DATA 


Range of 
Difficulty 


Discrimination 


Revision 


Test 
Item 


Material 


90-100 


+ 


No 
No 


No 
No 


No 
No 




0 


No 


No 


No 


80-90 


+ 


No 
Yes 


No 

• 


No 

7 

• 




0 


No 


No 


No 


60-80 


+ 

0 


Yes 
Yes 
Yes 


7 

Yes 

7 

• 


Yes 
Yes 

7 


0-60 


+ 

0 


Yes 
Yes 
Yes 


7 
• 

7 
• 

7 


7 

• 

7 
• 

7 



Since data on item discrimination was not available 
until late in the revision cycle, it was not possible to 
fully implement these decision rules and verify their 
usefulness in the revision process. However, in 
subsequent guidelines to writers. 

Revision of progress checks was based on the 801 
difficulty level, the item discrimination index, and 
the item response analysis. Since these procedures 
have already been indicated above, this section is 
limited to revision procedures for the administrative 
and cumulative posttests. 

28?; 
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Administrative Test Revision 

The following guidelines were given to writers 
for the first revision of administrative test items: 

1) Identify and revise all items which are above 
7SI difficulty on the pretest. 

2) Identify and revise all items which have less 
than 75% difficulty if the difficulty of the 
corresponding posttest item is less than 201 
above that of the pretest item. 

3) Identify and revise items if the posttest 
difficulty is less than 701. 

4) Under condition 2 above, revise all items 
which have negative discrimination indices or 
which have positive discrimination indices 
that are significant. 

5) Wherever possible, revise administrative items 
at the same time the corresponding materials 
and progress check items are revised, and 
revise materials if they are the cause of 
negative discrimination or low difficulty. 

After all administrative items had been revised and 
were reviewed by WLC subject matter experts, the items 
were pretested on two groups of USNA midshipmen, plebes 
and second classmen. Plebes were tested in order to 
determine which items were simply too easy. Second 
classmen were tested in order to determine which items 
measured objectives that second classmen had already 




Specifications given to writers for the second 
revision of administrative test items were: 

1) Work from second class to plebe data. 

Identify all items which 751 or more second 
classmen answered correctly. If 501 or more 
plebes also answered correctly, it is simply 
too easy and needs to be revised or replaced. 
If less than 501 of the plebes answered correctly, 
correctly, leave it alone unless 801 or more of 
of the second classmen answered correctly. 
In the latter case, replace the item. 
21 Work from plebe to second class data. Identify 
all items which 501 or more plebes answered 
correctly. If 66% or more second classmen 
also answered correctly, it is too easy and 
needs revision. 
3) Use confidence data where it appears appropriate, 
i.e., if in doubt as to necessity for revision, 
check the confidence data for second classmen. 
The percentage of students getting the item 
correct but indicated no confidence in their 
response is the percentage who were guessing. 
However, confidence data should in no way 
suggest necessity for revision which the item 
response data does not suggest. 
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Cumulative Posttest Revision 

Since CPT items were constructed to discriminate 
among students and among modules and since there was 
no pretest for CPTs, the guidelines for revision of 
CPTs differed from those for the administrative tests. 
Guidelines for CPT revision were: 

1) Identify all items which are over 8S% difficulty 
or below 405 difficulty. These items will 
usually need to be revised considerably or 
replaced. 

2) Identify items which are between 75% - 85% 
difficulty if they contain one distractor which 
has elicited no responses, and revise that 
distractor. 

3) Identify items which are between 65% - 75| 
difficulty if they contain two distractors 
vfhich elicit no responses, and revise either 
of the distractors. 

4) Identify items between 40% - 501 if they 
contain a distractor which elicits more correct 
responses than the actual correct response, 
and revise either the distractor or the 
correct alternatives. 

5) Identify and revise items between the 40% - 80% 
difficulty level that are negatively 
discriminating and were not revised according 
to the difficulty rules. 

3{!3 
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6) If an item has a significant positive 
discrimination index, do not revise it. 



The foregoing itemization of test item development 
procedures, from the point of view of the development model, 
must necessarily be considered in relationship to the manage 
ment functions for. the total process. Considering the large 
number of test items which were required for such a project 
the logistics involved in developing them was rather complex 
The specific problem, forseen well in advance was the need 
for articulation between test item^evelopment and the other 
products being prepared for student use... Because of the 
press of time many of the items had to be 'generated at least 
in draft format by consultants who were unfamiliar with the 
intricacies of the project. While these consultants were 
furnished copies of the content outline and the terminal and 
enabling objectives, in addition to specifications for test 
item construction, experience had indicated that the very 
nature of this course increased the likelihood that inter- 
pretations for content in "soft science" could vary. Thus 
mechanisms were set up whereby technical assistance to the 

test item developers was available on a round-the-clock 
basis. The consultants were encouraged, indeed ordered, to 
call for technical direction on any point which was open to 
interpretation. When the test items were delivered, they 
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were carefully reviewed by in-h.ouse personnel. Thus 
continuity and congruence between all materials and test 
items were maintained. 

The number of test items generated, as determined 
by contract, was to be in number 2001 over the number of 
objectives. Since the number of objectives totaled al- 
most 2,000 enabling objectives, a compromise had to be 
reached whereby a workable number could be specified for 
test production. The arrangement that evolved and was 
agreed upon was that 24 test items would be genera ted for 
every segment of instruction. These items would constitute 
the test item pool from which could be drawn an appropri- 
ate selection for the various test. Additionally research 
test items were generated, approximately 600 in number, 
which would be later incorporated into the test item pool. 
Thus for this project over 2,000 test items were prepared 
in addition to those embedded in the instructional mater- 
ials themselves. Because much of the material was pro- 
grammed, and had as a minimum two questions per page» 
these numbered in excess of 10,000. 
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This section of the model for course development 
has concentrated on test development and validation. 
Some o£ this process will be continued in the next chapter 
which represents the final stage of the model - namely, 
revision. The specific task at hand was to prepare a 
series of evaluative measures for insuring the performance 
of the materials and to gain some insight, into the rela- 
tive performance of varying learning styles. While a great 
deal of discussion centered on the best mechanism for 
achieving those goals it can be said that the orientation 
was to a practical rather than a theoretical consideration. 
Further details on the development and rationale for re- 
search and evaluation is available in Part I of this final 
report (TR 6.18), 
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VII. REVISION PROCEDURE 



Abstract 



This section effectively completes 
the model by discussion of those 
procedures and methodologies used to 
refine the performance of instruc- 
tional materials. It cites examples 
of data collected and their use in 
upgrading effectiveness. 
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VII. REVISION PROCEDURE 

This section of the Final Report is not intended to 
outline the rigorous details for evaluation and research 
procedures which are reported in Part 1. There is, 
however, a requirement for the developmental model to 
describe the procedures and rationale for revision and 
subsequent production that took place. Furthermore, 
these procedures and rationale which were reviewed by 
the customer before implementation helped to allow a 
smooth transition from phase to phase. Objections, res- 
ervations, or additions were usually specified in writing 
so that there would be no misunderstanding as to customer's 
priorities . 

At discrete points, and at least at the end of each 
phase, a summary of activities and accomplishments completed 
was listed. For example, the following seemed significant 
at the end of Phase II, and were noted. 

1. Development and revision of content; outlines 
and objectives. 

2. Development of course strategy and presentation 
design. 

3. Development and preliminary revision of course 
materials for 59 core segments in 8 media. 

4. Development- and preliminary revision of progress 
checks. 

293 
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5. Development and preliminary revision o£ research 
tests for 16 units. 

6. Development o£ five experimental research, 
hypotheses. 

7. Development of evaluation techniques. 

8. Development of data collection and processing 
procedures. 

9. Development of course logistics, 

10. Course implementation with 50 midshipmen. 

Evaluation data collected fell into objective and 
subjective categories. 
Data and Revision' Flow 

Objective data for course revision was compiled on 
the following variables: 

1. Administrative pre-, and post-tests. 

2. Cumulative post- tests. 

3. Progress checks. 

4. Student confidence ratings. 

5. Number of remediations . 

6. Questionnaires. 

7. Student time. 

8. Module frames. 

Most of the compilation of data on these variables were 
done via computer at Measurement Research Center, Iowa City, 
Iowa. Certain tabulations, however, were performed 
concurrently by hand at Annapolis. It was convenient to 
depict the flow of data in chart form, 
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All data were first checked in-house in order to 
insure the student's accuracy in completing tests 
and forms. Preliminary item difficulty was also 
tabulated at WLC Annapolis for cumulative post test 
in order to find and delete faulty items which might 
affect experimental results. CPT items were then 
re-checked by the Research staff and mailed directly 
to consultants for analysis of variance within experimental 
hypotheses. 

Test Reports and Statistical Summaries 

In all cases, test results for Administrative 
tests, CPT's, and progress checks were sent to Measurement 
Research Center in Iowa for recording of data by student, 
for cietialed item analysis, and for summarizing and 
reporting data back to WLC. After the summary data had 
been edited within the data files, the edited summaries 
were reported to the WLC research staff and from there 
were disseminated to' appropriate persons in either the 
materials preparation department, or the customer course 
administration staff. 

Summary progress check data generated by Measurement 
Research Center were reported in the form given on p:;ge 503. 

It can be noted from the sample form that 721 of the • 
students achieved at least 801 of the objectives on 
their first try. The mean score for the first try was 
8.1 on the 10 point test and the average confidence was 
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7.3. Combining the number of students on the second 
iteration who scored 805; or better after remediation 
*: with those from the first, it can be noted that 36+ 7 » 

43 or 861 of the students achieved 80-^ or more of the 
objectives. In the sample cited, test items relate 
directly to objectives and can therefore be viewed as 
% of objectives attained or failed. 

The summary analysis of progress checks was also 
repeated by module for segments contained within the 
research design. In other words, it was possible to 
assess the relative effectiveness of single modules by 
applying the I - % criterion to each module. A difficulty 
with reporting percentages for modules, however, was the 
crudeness of percentages when accumulated on small 
numbers . 

The percent of students attaining a specific 
objective throughout the course was reported in the 
form of summaries of item difficulty tabulations with 
respect to corresponding objective references. An 
example of the test item objective matrix is presented 
on the next pages. 
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S U M M A R Y A N A L Y S I S 
Progress Checks 
PART ONE, SEGMENT I Number of Items = 10 

; Iteration 1 Iteration - 2 

4 



Percentage 
Correct 


FD 
PC 


Average 
Confidence 


Percentage 
Correct 


FD 
PC 


Average 

P nn *r ■? H ATI PA 


lOOxxx 


^14 .0 


7 6 


J. V U AAA 


A. m ^ M 

*40 .0 


O * O 


90-99 


1428.0 


7.3 


90-99 


^10.0 


9.0 


80-89 


1^30.0 


7.2 


80-89 


^20.0 


8.2 


70-79 


9l8.0 


7.5 


70-79 


^20.0 


8.1 


60-69 


36.0 


7.1 


60.69 


iio.o 


8.6 


50-59 


^2.0 


8.0 


50-59 


Oo.o 


0.0 


40-49 


%.o 


0.0 


40-49 


%.o 


0.0 


30-39 


^2.0 


6.4 


30-39 


Oo.o 


0.0 


20-29 


Oo.o 


0.0 


20-29 


Oo.o 


0.0 


10-19 


Oo.o 


0.0 


10-19 


Oo.o 


0.0 


0-9 


Oo.o 


0.0 


0-9 


Oo.o 


0.0 



Total NCT = 50 
Mean Score = 8.1 
Mean Confidence = 7.3 
72% scored 801 or better 



Total NCT = 10 
Mean Score = 8.5 
Mean Confidence = 8.4 
70?; scored 80% or better 
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SUMMARY ANALYSIS 
TEST ITEM - OBJECTIVE MATRIX 
Part - 1 Segment - 1 Number of Items = 10 
Number of Objectives = 10 

Iteration - 1 



Objective 
Reference 



Test 
Item 
Number 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 



1 

80 



8 



10 



70 



80 



80 



60 



70 



80 



70 



60 



80 
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Item discrimination was assessed by rank bi-serial 
correlations o£ total test score rank equivalents and 
item correct-incorrect responses. On clarification o£ 
the correlation procedures, the format for item difficul 
presented was modified to include the discrimination 
index for each item and the level of significance of 
the discrimination. 

Item analysis data was used in making decisions 
concerning the revision of test items, the revision of 
materials, or both. 

A fourth tyje of summary data was the tracking of 
individual students throughout the course. The sample 
format for the individual track is given. 

SUMMARY DATA 
STUDENT TRACK 

Student Number 
Research Number 
Student Name 

Administrative Pre Test - 37 

Iteration 1 Iteration 2 
Part-1 Segment- 1 Module-1 
Part-1 Segment-2 Module-1 
Part- 2 Segment- 1 Module-1 
Part- 2 Segment-2 Module- 3 
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From the test item-objective matrix, it was a 
simple process for course writers to determine the 
exact objectives which were not being clearly presented 
or tested. 

A third type o£ summary data provided by iMeasure- 
ment Research Center was the item analysis summary for 
all test items included in the course. Item analysis 
included item difficulty, distributions of confidence 
ratings, and item discrimination. A sample summary 
format for item difficulty is given below. 

PROGRESS CHECK ITEM ANALYSIS 

PART 3 SEGMENT 3 ITERATION 1 NO. ITEMS = 10 
ITEM I.D. ITEM RESPONSE 







FD 


PC 


3401407 


A 


0 


0.0 




*B 


44 


95.7 




C 


1 


2.2 




D 


1 


2.2 


NCOUNTS 




46 





(The distribution of confidence ratings is also 
accumulated for each alternative.) 
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After the student track summaries had been reviewed 
by the research staff and the data files had been edited, 
student track summaries were forwarded to the USNA. It 
was not at all expected, however, that a student's 



portion of his grade. 

Subjective data was compiled from three major sources. 

1. The student. 

2. The on-site instructor and other members of the 
USNA leadership committee. 

3. The WLC course administrators and research staff. 
Student-Subjective Data 

* 

Subjective student data came in the form of unsolicited 
student comments, complaints or suggestions, and were 
considered extremely valuable sources of information for 
on-the-spot deletion or revision of test items. For 
example, of the more than 1,000 test items which had been 
developed, in some few cases students had disagreed with 
the designated correct answers for items. In these cases, 
course administrators reviewed the items and made correc- 
tions as necessary. Student comments concerning poor test 
items pre-empted ordinary item analysis compiled after all 
students had answered the item. The students were highly 
familiar with the materials at the time of the progress 
check and were able to indicate weaknesses of the items 



cumulative scores on progress checks would comprise any 
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perhaps more efficiently than a test item writer. Faulty- 
items had been noted according to student comments, and 
on-going revisions to items were documented. 

Student comments and suggestions concerning entire 
segments of content, material, and forms of presentation 
were also noted and used in conjunction with hard data 
as well as other forms of subjective data in reviewing 
and revising total course materials. 

A second important source of subjective data from 
which information was compiled and studied was that pro- 
vided by the USNA on-site instructor and members of the 
Leadership Committee. Input from all professionals in 
the study of Leadership had, in the past, been used 
primarily in developing and sequencing the objectives and 
content for the course, and in reviewing materials and tes 
items for their correspondence to objectives and content 
outlines. This function of customer leadership groups 
was still considered important to the extent that there 
were residual inconsistencies in the content and objec- 
tives, but was not expected that there would be input 
effecting appreciable changes in either content or 
objectives. In other words, it had been assumed that 
the direction, substance and detail of the Leadership 
course had been finalized. 
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Further subjective imput was to come from the review 
o£ revised materials. Also there was some subjective 
evaluation of the implementation of the course. The 
evaluation was made with respect to the logistical ease 
of implementation and the amount of time required for 
students to adequately achieve the objectives of the 
course. For example, if it was felt that students in 
Leadership were spending a disproportionate amount of 
time studying Leadership, in contrast to the time spent 
in other subjects, this factor would have to be weighed 
heavily in the design of the final course product. 

Another source of subjective input bearing on course 
revision was the WLC course administration and research 

staff. In light of the fact that the first and second 
implementations of the course were designed to provide 
information for a tightly constructed experimental design, 
it was noted that there were many artificial constraints 
on both students and presentation forms which were then 
included in the course. Decisions were to come from an 
interpretation of total findings with respect to both 
final design and logistical administration. 
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Revision after the first implementation, a validation 
test run, set the procedures for subsequent revisions. 
In summary, specific activities leading to the first 
implementation were: 

1. Specific content for the course was identified. 

2. Behavorial objectives were written based on the 
desired content. 

3. Behavorial objectives were sequenced by part and 
segment. 

4* Content outlines w» re expanded. 

5. Behavorial objectives and content outlines 
were revised. 

6. Research experiments were identified. 

7. Presentation designs for each part and segment 
were identified (media and presentation forms 
for both research and nonresearch segments), 

8. Materials were developed according to the 
presentation design. 

9. Test items were developed for all segments. 

10. Content, objectives, materials, and test items were 
revised according to United States Naval Academy 
recommendations . 

11. First implementation began (with 50 United States 
Naval Academy midshipmen) . 

The actual development of the course did not proceed 
as smoothly step to step as the above outline suggests. 
It can be noted that there were several revision points 
prior to the first implementation since content outlines 
and behavorial objectives were revised several times both 
before and during materials development • 
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Having determined that the content of the course 
was as desired, the adequacy of t>.e materials and test 
items covering the content beca. the relevant consi- 
deration. In order to assess th<^ adequacy of materials 
and test items and to make appropriate revisions to 
both, WLC developed the evaluation and revision 
procedures which are presented in the following 
sections • 

IDENTIFICATION OF SEGMENTS FOR REVISION 

The first step in the revision cycle was to collect 
all data relevant to the identification of segments 
requiring revision. Data pooling into segment identi- 
fication are: 

a. objective data on segment % - % effectiveness 

b. objective data on module I - S effectiveness 

c. objective data from module questionnaires 

d. subjective data by students 

e. subjective data by USNA on-site instructors 
and members of the Leadership committee 

f . subjective data by WLC course administrators 

g. subjective data by WLC research staff 

The first set of data to be considered was the 
summary analyses of progress check results. From 
the summary analyses, all segments in which 801 of 



the students have failed to meet SO^ o£ the objectives 
were identified. 

Materials in segments which did not meet the 801 
80% criterion were reviewed and revised. Segments 
ineeting the 801 - 80% criterion were reviewed but were 
not expected to be revised to the same extent. 

Having identified segments which fell below 
criterion, the second step was to observe % - I 
achievement for modules contained within faulty 
segments. By this method it was possible to note 
that some modules were quite effective, while other 
ineffective modules tended to lower the % correct for 
the entire segment. In such cases, ineffective module 
were given priority in revision. The research staff 
was consulted before revisions were made to modules 
which could affect the experimental treatments in the 
research hypotheses. 

Information from module questionnaires was used 
as supplementary information for screening segments 
requiring revision. Answers to questions such as "was 
the material interesting," "was the material difficult 
and "were the questions difficult" were scanned for 
discernible patterns. 

Subjective data from students, such as complaints 




s 



I 



I 
I 
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comments, and suggestions were helpful in isolating 
weak segments. Students are invited to comment on 
segments in writing at the end o£ the module questionnaire, 
Additionally, the WLC course administrators kept note!=i 
on student comments (oral communications) which related 
to segments or portions o£ segments o£ instruction. 
In segments where students performed at an acceptable 
level but were unhappy about the materials, objective 
data was weighed more heavily. However, for segments 
which fell below criterion effectiveness, student 
comments helped to identify general problem areas. 
Following identification of segments needing 
revision, WLC materials revision personnel began 
identifying the specific test items or portions of 
material which required revision. Identification of 
faulty test items and materials was based on the 
following sources of data. 

a. the test item-objective matrix 

b. item difficulty 

c. item discrimination 

d. item confidence ratings 

e. student reports by WLC course administrators 

f. frame analysis 

Test Item Objective Matrix 

Analysis of the test item-objective matrix yielded 



313 



3?8 



information about objectives which had not been passed 
by the criterion percentage o£ students. Since all 
test items were keyed directly to objectives, the 
writer's task of isolating the material leading to the 
test item was greatly simplified. That is, the writer 
could refer to the specif: c content covered by an 
objective oi objective and review the correspondence 
among content, objectives anr tesr items. By this 
process, the writer might have noted that there was 
little real relationship between the three and then 
noted the need for sharpening the congruity among them. 

In cases where the test item and objective did 
not correspond, the test item was revised in order 
to reflect the intent of the objective. If the 
objective and content (material) did not correspond, 
the materials were revised to adequately cover the 
intended objective. 

To the extent that materials, objectives, and 
test items all appeared to be related to each other, 
yet students were failing to meet objectives, analysis 
was first to be made of the adequacy of the test items. 
The first source of objective data on test items 
was the item difficulty index and distribution of 
student responses over all test item alternatives. 
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If the difficulty of the test itei?* was high, (i.e., 
if over 60v of the students answered the item correctly 
and responses were equally distributed over the 
three remaining alternatives) the test item itself 
was considered adequate. In such a case, it was 
reasoned that if 60% or more of the students were 
able to identify the correct answer and no other alterna- 
tive appeared equally reasonable, then the test item 
itself was probably properly constructed. The next 
step was to review materials covered by the test item. 

If the item was found to be of high difficulty 
with equal response distribution and the item discrimi- 
nation index v/as positive, ther it was possible that 
differences in module presentation account for the 
lack of percentage correct performance. In other 
words, it could be that some modules covered the 
materials related to an objective more effectively 
than others. In cases of nonresearch segmc^nts the 
identification of test items or materials to be revised 
was not so clear-cut. 

If the difficulty of the test item was high, 50^ - 
601, but not over 60?;, and the remainder of students 
all picked the same wrong alternative, then it was 
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possible that the test item: (1) included two correct 
responses, (2) had two entirely implausible alternatives 
with two entirely plausible alternatives, or (3) 
required too fine a discrimination. Materials were 
reviewed in the first case to decide if both answers 
were in fact correct or if wrong information was being 
presented as correct. Otherwise, the test item itself 
was revised. 

If the difficulty level of an item was loxv (i.e., 
if 40% or fewer students answered the item correctly, 
then both test items and materials were reviewed and 
revised as necessary. In cases of low difficulty, 
the correct answer may have been keyed as incorrect 
or a supposed distractor may actually have been a 
better choice of correct. If the writer was satisfied 
that the item was correct according to the objective 
it measured, then materials related to the objective 
were revised. 

Item Discrimination: 

The decisions based on discrimination indices 
were not intended to be inclusive o£ all possible 
combinations of item difficulty and item discrimination. 
They were rather intended as illustrations of the use of 
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objective test-item data in analysis o£ test items 
and materials for revision. In some cases, errors 
in test items and materials were obvious to a writer 
and he quickly corrected the problem. In other 
instances, even the item analysis did not help in 
pinpointing the difficulty and the writer was required 
to rewrite portions of materials plus develop new test 
items. Wherever available, pretest data 
on progress checks was used in diagnosing areas of 
difficulty in test items. 

USNA REVIEW 




Following the analysis of data and diagnosis of 



materialrB--tequiring revision, WLC submitted the revised 
segments for review->a.ccording to schedule. In submitting 
materials, WLC specified portiojis of materials which 
had undergone major revision. Since WLC was- 
working closely with in-house experts in revising 
materials, it was not expected that a large portion 
of the revision burden would be placed on the USNA. 
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This section of the report basically concludes 
discussion o£ tfue model. Materials revision preparatory 
to retesting closes tRe loop wKich began with definition 
of content. In many respects the process began again 
since revision necessitates a relook at the steps lead- 
ing up to development and student testing. Thus^ it is 
an iterative process where success is achieved through 
successive approximations towards the desired goals. 

Some final notes on the revision process may be in 
order here. One item concerned with both Chapters VI 
and VII is the possible determination of a single set 
of criteria for good test items* It was the experience 
of this project that no such generalized measures evolved. 
However, some general considerations were used which were 
most helpful and worthwhile. The first real evaluation 
of a test item was done by the subject matter expert to 
insure content validity. In other words did the test 
question accurately incorporate the desired content and 
was the answer specified as correct indeed the most ac- 
curate and appropriate possible. At the second stage 
since most questions were problem solving in nature , 
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was the description described including terminology used 
a realistic and accurate reflection of a potential leader- 
ship situation. There was little attempt prior to actual 
testing by the subject matter experts to ascertain other 
characteristics of the test items value. Statistical 
data would later provide the guidelines on levels of 
difficulty and discriminatory powers as detailed in both 
Chapters VI and VII. 

Late in the project some criticism arose over the 
possibility of some items being overcued. At that time 
all items in the progress check tests were examined to 
see if terminology was leading as to make the choice 
obvious, if there were certain simple placement patterns 
which would cue the student that certain choices could 
never be correct, or thirdly, if some items were made 
obvious by such devices as the length of the answer. In 
the process of this examination some items were changed 
while others proved to be quite respectable by virtue 
of frequency distributions recorded for those items. 

The following brief section completes this report 
with some general summary comments. 
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VIII. SUMMARY AND CONCLUSIONS 



Abstract 



This brief section summarizes the steps 
of the model , and briefly evaluates 
progress made and lessons learned. 
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Th.e project, therefore, successfully concluded, 
as the three year effort to advance the practical art 
of course development yielded both a viable and re- 
plicable model and a smoothly performing course system. 
The additional values that have accrued include the 
research and evaluation results, and the training of 
almost hundreds in the systems approach to course 
development procedures. This experience of an inter- 
disciplinary, systematic methodology has spin offs 
in giving the educational community a cadre of knowledg- 
able professionals, capable of making additional con- 
tributions to educational advancements. For if the 
educational systems for the 1970 's are to meet the 
needs that are now so desperately obvious, the people 
as well as the models are necessary. 

The research efforts were at first disappointing 
since the potential for macro specifications for 
materials development did not materialize. The "no 
significant differences" reports initially rippled 
through the project in an almost ominous manner. 
Outside critics of the project took the opportunity 
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to additionally finger th.e research, as a distractor 
from the main effort of course development. Again, 
the research was misunderstood as being superfluous 
to the project, when in reality- it was the contractor's 
effort to fulfill the requirement to study alternative 
methods of presentation. 

In further reflection, the "no significant 
differences" took a different meaning when evaluation 
data showed that overall performance of all methods 
of presentation suggested that the skill of materials 
preparation and revision tended to eliminate any 
differences that might otherwise show. Thus, as re- 
ported in PART ONE of this final report, one yield 
from the research effort points to the need for 
further research on the skills and techniques of the 
educational writer. Indeed the repertoire of skills 
and attitudes in the writer may be of more significance 
than the competancies of the learner. 

Overall, the project seems to conclude with more 
data of importance as developmental variables Ce.g. 
management and technical skills and attitudes) than 
as learner variables. While learner variables were 



studiously sought for, the results at this time were 
more interesting than practical for individualizing 
course instruction. Perhaps the difficulty lies with 
the present taxcnomies of variables that encompass 
skills that are tv.o generalized C^.g. SAT Verbal) or 
perhaps are totally irrelevant to the predispositions or 
individually acquired strategies for learning C^.g. 
agressiveness) . Only further study will answer that. 
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